BIOT: Biosignal Transformer for Cross-data Learning in the Wild
Authors: Chaoqi Yang
M Brandon Westover
Jimeng Sun
Venue: NeurIPS 2023 poster (2023)
Keywords: biological signal
transformer
cross-data learning
in-the-wild learning
TL;DR: This paper proposes a general biosignal transformer (named BIOT) that allows learning cross datasets with mismatched channels, variable lengths, and missing values.

Abstract: Biological signals, such as electroencephalograms (EEG), play a crucial role in numerous clinical applications, exhibiting diverse data formats and quality profiles. Current deep learning models for biosignals (based on CNN, RNN, and Transformers) are typically specialized for specific datasets and clinical settings, limiting their broader applicability. This paper explores the development of a flexible biosignal encoder architecture that can enable pre-training on multiple datasets and fine-tuned on downstream biosignal tasks with different formats.

To overcome the unique challenges associated with biosignals of various formats, such as mismatched channels, variable sample lengths, and prevalent missing val- ues, we propose Biosignal Transformer (BIOT). The proposed BIOT model can enable cross-data learning with mismatched channels, variable lengths, and missing values by tokenizing different biosignals into unified "sentences" structure. Specifically, we tokenize each channel separately into fixed-length segments containing local signal features and then rearrange the segments to form a long "sentence". Channel embeddings and relative position embeddings are added to each segment (viewed as "token") to preserve spatio-temporal features.

The BIOT model is versatile and applicable to various biosignal learning settings across different datasets, including joint pre-training for larger models. Comprehensive evaluations on EEG, electrocardiogram (ECG), and human activity sensory signals demonstrate that BIOT outperforms robust baselines in common settings and facilitates learning across multiple datasets with different formats. Using CHB-MIT seizure detection task as an example, our vanilla BIOT model shows 3% improvement over baselines in balanced accuracy, and the pre-trained BIOT models (optimized from other data sources) can further bring up to 4% improvements. Our repository is public at https://github.com/ycq091044/BIOT.

BIOT: Biosignal Transformer for Cross-data Learning in the Wild
Chaoqi Yang', M. Brandon Westover Jimeng Sun University of Illinois Urbana- Stbana-Champaign. 'Harvard Medical School Beth Israel Deaconess Medical Center
( edu
Abstract
Biological signals, such as electroenceehallgrams (EEG), play crucial role numerous clinical applications, exhibiting diverse data formats and quality profiles. Current deep learning models for biosignals (based on CNN, RNN, and Transformers) are typically specialized for specific datasets and clinical settings, limiting their broader applicability. This paper explores the development of a flexible biosignal encoder architecture that can enable pre-training on multiple datasets and fine-tuned on downstream biosignal tasks with different formats.
To overcome the unique challenges associated with biosignals of various formats, such as mismatched channels, variable sample lengths, and prevalent missing val- ues, propose Biosignal Transformer (BIOT). The proposed BIOT model enable cross-data learning with mismatched channels, variable lengths, and missing values by tokenizing different biosignals into unified "sentences" structure. Specifi- cally, we tokenize each channel separately into fixed-length segments containing local signal features and then re-arrange the segments form long "sentence" Channel emheddings and relative position embeddings are added to each segment (viewed "token") preserve spatio-temporal features.
The BIOT model versatile and applicable to various biosignal learning settings across different datasets, including joint pre-training for larger models. Compre- hensive evaluations on EEG, electrocardiogram (ECG), and human activity sensory signals demonstrate that BIOT outperforms robust baselines in common settings and facilitates learning across multiple datasets with different formats. Using CHB-MIT seizure detection task an example, our vanilla BIOT model shows 3% improvement over baselines in balanced accuracy, and the pre-trained BIOT models (optimized from other data sources) can further bring up to 4% improvements. Our repository is public at httpss//gthuub.oo///qpg/ 1044/BIOT
Introduction
Biosignals, such as EEG and ECG, are multi-channe chaanel time series recorded at high sampling rates (e.g.. 256Hz) various healthcare domains, including sleep medicine, neurological and cardiovascular disease detection, and activity monitoring. Deep learning (DL.) models have demonstrated impressive success in automating biosignal analysis across diverse applications (Yang et al. 202T), encompassing sleep stage classification (Biswal et al. 2018 Yang et al., 2021 Phan and Mikkelsen (022), emotion analysis via EEG (Zhang et al.) 2020 Suhaimi et al.) 2020). action and motor imagery recognition (Venkatachalam al. 2020), acute stress detection through electrodermal activity (Greco et al. 2021),
EEG-based seizure epilepsy classification (Yang et al. 2023 Jing et al. 2023). and ECG-driven cardiac arrhythmia detection (Isin and Ozdalili 2017: Parvanch et al. 2019).
Various deep learning methods have been developed for biosignal learning. Some works use ID convolutional neural networks (CNN) raw signals (Jing et al. 2023 Nagabushanam al. 2020
37th Conference on Neural Information Processing Systems (NcurIPS 2023).

---

Dar et al. 2020). while others preprocess the data with short-time time Fourier transform (STFT) and employ 2D CNN models on the resulting spectrogram (Yang al. 2022] Kim et al. 2020 Cui et al.) 2020). Researchers also segment the signal and use CNN segment encoder with downstream sequence model (Zhang et al. 2019 Biswal et al. 2018 Ding et al. 2020, Almutairi et al. 2021 , such as Transformer or recurrent neural networks (RNN), to capture temporal dynamics. Other approaches involve ensemble learning, feature fusion from multiple encoders (Li et al. (022). and multi-level
transformers to encode spatial and temporal features across and within different channels (Lawhern
ct al.
2018 Song et al.
2021
Liu et al. 202T).
These models (ing et al.
2023 Yang et al. 2021 Biswal et al. 2018 Kostas et al. 2021 Du et al.
2022 Zhang et al. 2022) predominantly focus on biosignal samples with fixed formats for specific tasks, while woald data may exhibit mismatched channels, variable lengths, and missing values. In this paper, our objective is to devise a flexible training strategy that can handle diverse biosignal datasets with varying channels, lengths, and levels of missingness. For example, is it possible to transfer knowledge from abnormal EEG detection (a binary classification task with samples having 64 channels, 5-second duration, recorded at 256Hz) to improve another EEG task, such as seizure type classification (a multi-class task with 16 channels and a 10-second duration at 200Hz)? In reality, such data mismatches often arise from varying devices, system errors, and recording limitations. Additionally, it is also important to explore the potential of utilizing various unlabeled data sources.
To apply existing deep learning models to such settings of different biosignals, significant data processing required to align the formats across multiple datasets. This may involve truncating or padding signals for consistent lengths (Zhang et al. 2022), and imputing missing channels or segments (Bahador et al. (202T). Such practices, however, may introduce unnecessary noise and shift data distributions, leading to poor generalization performance. Developing a flexible and unified model that accommodates biosignals with diverse formats can be advantageous.
In our paper, we develop the biosignal transformer (BIOT, summarized in Figure T]. which, to the best of our knowledge, is the first multi-channel time series learning model that can handle biosignals of various formats. Our motivation stems from the vision transformer (VIT) (Dosovitskiy et al. 2020) and the audio spectrogram transformer (AST) (Gong et al., (021). The ViT splits the image into "sentence" of patches for image representation while the AST splits the audio spectrogram into a "sentence" " for 1D audio representation. We are inspired by that these "sentence" structures combined with Transformer (Vaswani et al. 2017) can handle variable-sized inputs.
However, it is non-trivial to transform diverse biosignals of various formats into unified "sentence" structures. This paper proposes BIOT to solve the challenge by a novel biosignal tokenization module that segments each channel separately into tokens and then flattens the tokens form consistent biosignal "sentences" (illustrated in Figure 2h. With the design, our BIOT can enable the knowledge transfer cross different data in the wild and allow joint (pre-)traimingg )traininnig on multiple biosignal data sources seamlessly. Our contributions are listed below.
Biosignal transformer (BIOT). This paper proposes generic biosignal learning model BIOT by tokenizing biosignals of various formats into unified "sentences."
Knowledge transfer across different data. Our BIOT can enable joint ppraiitraining and knowl- edge transfer across different biosignal datasets in the wild, which could inspire the research of large foundation models for biosignals.
Strong empirical performance. We evaluate our BIOT on several unsupervised and supervised EEG, ECG, and human sensory datasets. Results show that BIOT outperforms baseline models and can utilize the models pre-trained on similar data of other formats to benefit the current task.
2 BIOT: Biosignal Transformer
As shown in Figure D our BIOT encoder cascades two modules: (i) the biosignal tokenization module that tokenizes an arbitrary biosignal (variable lengths, different channels, and missing values) into a "sentence" structure. This design can potentially enable previous language modeling techniques (Devlin et al., 2018 Liu et al.) 2019 OpenAl 2023) to empower the current biosignal models; (ii) a linear transformer module that captures complex token interactions within the "sentence" while maintaining linear complexity. After that, we also discuss the application of BIOT in different real-world settings. For all the notations used in the paper, we summarize them in Appendix A

---

Blosignal Transformer (BiOT) Encoder
canigra
Byyers
Lays spinuitation
Foumml tp 200md [ng.
Unserae-uttertion
DN 10
Thatering
1d 90
Fri K0
rermailatioo
ree
In d1
(r
Data preprocassing
Module Blosignal Tokenization
Module Transformer Encoding
Supervised Learning
[3) Unsupervised Pre-tralning
(4) Supervised Pre te-ttrainnng
S
S
Pp1
Fp2
Fp2
y
Q1
02
BOT
IIOTT
Busignal
Pongee
Transfer and Iear new channels
Nutomatically varisble length
(2) Supervised Learning (with missing)
Trandieee oar
Fp2
Pp
Y
Proddor
O4
Z
2
FA
Figure Biosignal Transformer (BIOT). (Upper) Given new data sample, we initially perform data preprocessing (resampling, normalization, tokenization, and flattening) to create a biosignal "sentence" using biosignal tokenization module. We then learn the complex interactions within the "sentence" through the linear transformer module. (Lower) BIOT encoder is versatile, enabling supervised learning on complete data, data with missing values, and pre-training and fine-tuning across diverse data formats and tasks.
2.1 Module 1: Biosignal Tokenization
The goal of this paper is to model heterogeneous biosignals (e.g., EEG samples with different channels for different tasks) with unified model. For example, common EEG samples (Lopez et al.] 2013), such as those for seizure detection, are recorded at 256Hz in the international 10-20 system for 10-second long (Klem et al. 1999). With standard 16 montage editing, the sample is essentially a multi-channel time-series, represented as a matrix of size (16, 2560). However, format mismatch may prevent the applications on other similar data, such different sampling rate (e.g.. 200Hz vs 256Hz) (Jing et al. (003)). mismatched channels (i.e., different datasets have their own novel channels), variable recording duration (i.c., 30s per sample vs. 10s) (Zhang al., (022). missing segments (i.e., part of the recording is damaged due to device error). Thus, existing models may fail to utilize the mismatched data from different datasets.
Motivation.
Our BIOT solves the above challenges of mismatches by the following steps. We assume one biosignal sample as S E R Here, I as the channels and J duration(s) x rate (H2) as the length.
Resampling, We first resample all data to the same rate, denoted by r E IR+ (such as 200Hz), by linear interpolation. The common rate could be selected following clinical knowledge of certain biosignal. For example, the highest frequency of interest in both EEG and ECG signals is commonly around 100 Hz, and thus 200 Hz 250 Hz can be suitable for typical EEG ECG applications, according to Nyquist-Shannon sampling theorem (Nyquist 1928 Shannon 1949).
Normalization: To alleviate the unit difference and amplitude mismatch across different channels and datasets, we use the 95- 95-percentile of the absolute amplitude to normalize each channel.
Formally, each channel S is normalized by percentile
S(i
[]]. 96%]
'htpss//en https//eew.wkkkppeiaaaarwwwv//// sysssmsssmmmmm (EEG)
3

---

Sample
Tokend
Fp1
Fp2
position emb
Sample 2:
Tokers
segment emb
Fp1
channel emb
01
pos on emb
Figure 2: Biosignal Tokenization (no segment overlap in the examples). Sample has four channels (Fpl, Fp2, 01, and 02) for seconds. We tokenize each channel into segments/tokens and then parameterize these 20 segments with three embeddings. On the right, we use different colors to represent the channels (blue-Fpl, brown-Fp2, green-Ol, and yellow-O2). Sample has mismatched channels (no 02), variable lengths (Fpl and Fp2 shorter and flipped), and missing values (in O1). Using our method, we can still tokenize Sample 2 in a comparable "sentence". On both samples, the peach orange area indicates the spatial- or remporal-relevatt tokens w.r.t. the highlighted token.
Tokenization: For handling length variation, we tokenize the recording of each channel into segments/tokens of length & E R and neighboring tokens can have overlaps of length p I (p < t, e.g., r and p = 0.5r corresponds to 1s and 0.5s). Thus, the k-th token (k: = 1, 2..... K) + t). The Here, the overlap p essential to maintain the temporal information for shorter signals. For example, the length of a signal is seconds (i.e., J = 3r), the set r, 0 only generates tokens per channel, while the missing values, we drop the corresponding tokens directly (as shown in Sample 2). Note that, our tokenization applies each channel, separately, which different from previous works (Almutairi ct al. 2021 Du et al. (022)) that split all channels together (which cannot work on Sample 2).
in the i-th channel can be represented by slicing S p) 1) number of tokens K per channel is limited by the inequality: (t p) 1)+
p) +
r,p 0.5r gives 5 tokens per channel (see Appendix C.3) In cases of
Flattening: We finally flatten tokens from all channels into a consistent "sentence".
The above steps are non-parametric. To study the effect of sampling rate r, token length 6, and overlap we provide ablation studies and insights in Appendix C.3 In the following, we design the token embedding for the biosignal "sentence". which combines information from three aspects.
Segment embedding. We leverage fast Fourier transform (FFT) to extract energy vector for each token where every dimension indicates the energy of a specific frequency. fully connected network (FCN) is then applied on the energy vector to transform it into the segment embedding. Directly modeling the raw time-series to learn a segment embedding can be another option.
Channel embedding (spatial). We learn embedding table for all different channels and add the corresponding channel embedding to the token. Each color represents one channel in Figure 2
Positional embedding (temporal). In biosignals, the token order within the channel captures temporal information. We thus add relative positional embedding to the token embedding by using the sinusoidal/cosine functions /Vaswani et al. 2017). which does not need learnable parameters.
The above three embeddings are summed create the token embedding, which can effectively capture the segment features well as the spatio-tempory features. Finally, we denote the tokenzied "sentence" as X Nx4 where N is the number of tokens and 4 is the embedding dimension.
2.2 Module 2: Linear Transformer
Transformer with linear complexity for long biosignal "sentence". Next, we want to leverage the Transformer model (Vaswani et al. (207)) for modeling the "biosignal sentence". However, our transformation may lead to long "sentences" due multiple channels. For example, the "sentence"

---

of a 64-channel EEG signal for 20 seconds (without overlaps) may have 64 x 20 = 1280 tokens, and longer with the overlaps. Given that the original Transformer model is known have quadratic complexity in both time and space, we adopt the linear attention mechanism (Wang et al. 2020 Katharopoulos et al. 2020) for biosignal learning applications.
Formally, 1et us assume WK W WO E glixi be the key, value, and query matrices. Our self-attention module uses a rank-d approximation for the softmax attention (NV N) by reduced-rank parameter matrices E E RNxdd F E RdxN (where d < N). The output H E IR xla is,
H Attention(XW) EXWX FXW")
(1)
(XWX
softmax
FXWV dxlz
(2)
VT2
Nxd
Main components of this module include one linear self-attention layer and one fully connected
network. To enable stable training. we add layer normalization (Ba et al. 2016) before each component, residual connection (He et al. 2016) after each component, and dropout (Srivastava et al. 2014) right after the self-attention (see Figure which improves the convergence.
BIOT Encoder. An illustration of our proposed BIOT encoder shown in Figure I (upper), which comprises the biosignal tokenization module and multiple blocks of linear transformer modules. We obtain the final biosignal "sentence" " embedding by a mean pooling step over all tokens. Note that appending classification [CLS] token at the beginning of the "sentence" (after Module 1) is also common option. However, we find it yields a slightly worse performance in our application, and thus we use mean pooling in the experiments.
2.3 Biosignal Learning in the Wild
Our proposed BIOT encoder can be applied in various real-world biosignal applications (illustrated in the lower part of Figure D These applications include (1) standard supervised learning, (2) learning with missing channels or segments, (3) & (4) pre-training on one or more datasets, and fine-tuning on other similar datasets with different input formats.
(1) Supervised Learning is the most common setting in the previous literature (Ting al. 2023 Biswal et al.) (2018). With the BIOT encoder model, we finally apply an exponential linear unit (ELU) activation (Clevert et al.) 2015) and linear layer for classification tasks.
(2) Supervised Learning (with missing). Many real biosignal data have mismatched channels. missing segments, and variable lengths, which prevents the applications of existing models (Jing) et al. 2023 Song et al. 202T). Flexible as our model is, BIOT can be applied in this setting using the same model structure as in (1).
(3) Unsupervised Pre-training, We can jointly pre-train a general-purposs purpose BIOT encoder on multiple large unlabeled datasets. In the experiments, we pre-train unsupervised encoder using 5 million resting EEG samples (16 channels, 10s, 200Hz) and 5 million sleep EEG samples (2 channels, 30s, 125Hz), which is later utilized to improve various downstream tasks.
For the unsupervised pre-training, we take the following steps (a diagram is shown in Figure I
Assume S is the original biosignal. We first randomly dropout part of its channels and dropout part of the tokens from the remaining channels, resulting in perturbed signal S.
We then obtain the embeddings of S and S by the same BIOT encoder. To form the objective, want to use the perturbed signal to predict the embedding of the original signal. Thus, an additioanl predictor (i.e., two-layer neural network) is appended for the perturbed signal following (Grill et al. 2020). We use Z and Z to denote the real embedding of S and predieted embedding from S.
Z (S), =
predictor(BIOT(S))
Finally, contrastive loss He et
2020 Chen et al. 2020 used on Z and Z to form the objective.
L =
CrossE
[softmax ((Z, Z )/T )
Here, T represents the temperature (T = 0.2 throughout the paper) and I is an identity matrix. In the implementation, we also apply sample-wise L2-normalization on Z and Z before matrix product.
5

---

(4) Supervised Pre-training aims to pre-train model by supervised learning on one task and then generalize and fine-tune the encoder on new task. The goal here is transfer knowledge among different datasets and gain improvements on the new task compared to training from scratch. Our BIOT model allows the new datasets to have mismatched channels and different lengths.
3 Experiments
This section shows the strong performance of BIOT on several EEG, ECG, and human sensory datasets. Section 3.2 3.3 compare BIOT with baselines on supervised learning and learning with missing settings. Section 3.4 3.5 3.6 show the that BIOT can be flexibly pre-trained on various datasets (supervised unsupervised) improve new task with different sample formats.
3.1 Experimental Setups
Biosignal Datasets. We consider the following datasets in the evaluation: (i) SHHS (hang et al.
2018 Quan et al.] 1997) is a large sleep EEG corpus from patients aged 40 years and older. (ii) PREST is a large unlabeled proprietary resting EEG dataset; (iii) Cardiology (Alday et al. (020) is a collection of five ECG datasets (initially contains six, and we exclude the PTB-XL introduced below). (iv) The CHB-MIT database (Shoeb) 2009) is collected from pediatric patients for epilepsy seizure detection. (v) IIC Seizure dataset is from Ge et al. (202T): Jing et al. (2023) for detecting one of the six ocaal-imerrittal-njjyy-ccoom continnuuuuuomuuum (IIC) seizure patterns (OTH. E L.PD, GPD, LRDA, GRDA); (vi) TUH Abnormal EEG Corpus (TUAB) (Lopez et al.) 2015) is an EEG dataset that has been annotated as normal or abnormal; (vii) TUH EEG Events (TUEV) (Harati et al. 2015) is a corpus of EEG that contains annotations of segments as one of six sleep or resting event types: spike and sharp wave (SPSW), generalized periodic epileptiform discharges (GPED), periodic lateralized epileptiform discharges (PLED), eye movement (EYEM), artifact (ARTF) and background (BCKG); (viii PTB- XL (Wagner al. 2020) is ECG dataset with 12 12-lead recordings for diagnosis prediction, and we used it for arrhythmias phenotyping in this paper; (ix) HAR (Anguita et al. 2013) is a human action recognition dataset using smartphone accelerometer and gyroscope data.
Table 1: Dataset Statistics
Datasets
Type (subtype)
Recordings
Rate
Channels
Duration
Sample
Tasks
SHHS
BBG (slcop)
5,445
125Hz
C3-A2, Cl-Al
30 scconds
5,093 522
Unsupervised pre-training
PREST
EDG (resting)
6.478
200012
16 montages
secunds
5,110.992
Unsupervised training Unsupervised pre-training
Candiology
BCG
21,264
500Hz
6or BCG leads
seconds
495,9700
CHB- MIT
EEG (resting)
686
256H2
16 montages
10 seconds
326,993
Binary (scizure or not)
IIC Seizure
EDG (resting)
2,702
20081
16 montages
secunds
165,309
Multi-class (6 seizure types)
TUAB
BBG (unknowe))
2,389
256Hz
16 montages
seconds
409.455
Binary (abwormal or not)
TUEV
EEG (sleep and resting)
11.914
256812
16 montages
5 scconds
112,491
Multi-clas class (6 evcal types)
PTB-XL HAR
ECG
21,911
500HL
ECG leads
sccondss
65.511
Binary (arryy or not)
Wearable sensors
10.299
5081z
9 cocrdinates
2.56 seconds
10.299
Multi class actions)
Dataset Processing, The first three datasets are used entirely for unsupervised pre- pre-training. The next four datasets are used for supervised learning, and we used the common 16 bipolar montage channels in the international 10-20 system. For CHB-MIT (containing 23 patients), we first use patient to 19 for training, 20,21 for validation, and 22,23 for test. Then, we flip the validation and test sets and conduct the experiments again, and we report the average performance on these two settings. For IIIC seizure, we divide patient groups into aaining/validationnness sets by 60%:20%::00% For TUAB and TUEV, the training and test separation is provided by the dataset. We further divide the training patients into training and validation groups by 80%:20%. For PTB-XL, we divide patient groups into sets by 80%:1 00%: 10%. The train and test set of HAR is provided, and we further divide the test patients into validation/test by 50%:50%. For all the datasets, after assigning the patients to either training, validation, or test groups, we will further split the patient's recording to samples, and the sample duration accords to the annotation files. The dataset statistics can be found in Table ① and we provides more descriptions and processing details in Appendix B.I
training/validation/ee
We consider the following representative models: (i) SPaRCNet (Ting et al. 2023 is a ID-CNN based model with dense residual connections, more advanced than the popular ConvNet (Schirrmeister et al., 2017), CSCM (Sakhavi et al., 2018); (ii) Contra WR's (Yang et al. 2021 encoder model first transforms the biosignals into multi-channel spectrogram and then uses 2D-CNN
Baseline.
based ResNet (He et al. 2016); (iii) CNN-Transformer (Peh et al., (022) is superior to CNN-LSTM
models (Zhang et al. 2019); (iv) FFCL (Li et al. 2022) combines embeddings from CNN and LSTM

---

encoders for feature fusion; (v) ST-Transformer Song et al. (2021) proposes an multi-level EEG transformer for learning spatial (S) and temporal (T) features tanuoune. empirically better than CeGGNetLLawhen et al. 20188. Our BIOT model trained from scratch is denoted by (vanilla).
Environments and Settings. The experiments are implemented by Python 3.9.12, Torch 1.13. 113.1+cul11 17, Pytorch-lightning 1.6.4 on a Linux server with 512 GB memory, 128-core CPUs and eight RTX A6000 GPUs. All the models are optimized on training set and evaluated on the test set. The best model and hyperparameter combinations are selected based on the validation set. For Table 2 and Table 3 we obtain five sets of results with different random seeds and report the mean and standard deviation values. For Figure andd Figure 4 we report the results under three random seeds. More experimental and implementation details can refer to Appendix B.2
3.2 Setting (1) standard supervised learning
This section shows that BICT is comparable or better than baselines in the supervised learning settings.
Four EEG Tasks. Both CHB-MIT and TUAB are designed to predict binary output, and we use binary cross entropy (BCE) for TUAB and the focal loss (Lin et al. (017)) for CHB-MIT due to its imbalances (around 0.6% positive ratio in training set). We use balanced accuracy (Balanced Acc.), area under precision-recall curve (AUC-PR) and AUROC the metrics. Both IIIC Seizure and TUEV are multi-class classification tasks with cross entropy loss. We employ Balanced Acc., Cohen's 's Kappa, and Weighted Fl as the multi-class evaluation. To save space, we only show the performance on CHB-MIT and IIIC Seizure in Table 2 and move the other two to Appendix C.I
ECG and Sensory Tasks. PTB-XL is formulated a binary classification on detecting arrhythmias phenotypes. We use the BCE loss and binary evaluation metrics. HAR (classifying actions) uses the cross entropy loss and is evaluated by multi-class metrics. Results are reported in Table We provide the running time comparison of all experiments in Appendix C.4
Table 2 and show that our model has superior performance over baselines in most tasks, especially on CHB-MIT, IIIC Seizure, and HAR. The reason might be that the frequency features are more useful in these three tasks as our BIOT extracts the main features from spectral perspective. SPaRCNet is a strong model among all the baselines except on the CHB-MIT task. The model might be vulnerable in the imbalanced classification setting even with the focal loss. The pre-training models at the end of
the tables will be introduced and explained in Section 3.4 3.6
Table 2: EEG classification tasks (Results of TUAB and TUEV are in Appendix C.I
Models
CHB-MIT (seinure defection)
Seizore (scizure type claciifiatioo)
Balanced Ape.
AUC-PR
AUBOC
Rolanced Ace
Cohos's Kappa
Weightod F1
SPaRCINer
2003
00191
0:0119
0.8143
0.0148
0.5546 0.0161
0.4679 0.0225
0.5569 00.084
ContraWR Ying 1r 2001
0.6344 0.0002
0.2261
0:0174
0,8097
0.0114
0,5519 0,0058
0:4623 0.0145
0.5886 0.0133
CNN-TTaus Cen-TTrsseemms
coma Perer
2022
B.6399 000667
02249
= 0:0227
0,8662
: 0.0002
0.5476
0.0033
0.4411
= 0.0139
0.5366 =: 0.0127
FFCT.
al 20-c
0.622 00104
0.2049
0:0346
0.8271
+ +.00051
0.5617
0.0117
D.704
+ 0.0330
0.5617 + 0.0171
ST Tratsrommn 200g at 2021
0
0.0195
0.1422
0.0094
0,8237
1 0.0099
0,522
0,0056
0,4492
1 0.0056
0,5400 1: 0.0014
(Vanilla) B10T
0,6640
0.0037
0.2573 1 0.0088
0,8646
1 0.0030
0.0034 0.5762
0,4932
1 0.0046
0.5773 1 0.0031
Pietmined B107 (PREST)
0.6922 + 0.0431
0.3072
0,1187
0,8679 + 0.0106
0.5787
0,0666
0.4900
+ 0.0054
0,8228
0.0049
Protmined BI0T PREST+SHISS
0.6788
0.036
0 000000
0:0003
0.8752
1 0.0022
0.5800
0.0004
0.5040 : 0.0041
0.5878
0.0015
Pretrained B107 EBG danasets)
0,706%
0.947
0.3277
0.8761
0.0244
0.5779
0.007
0,4949
0.0105
0.5557
0.0088
T.. All models use the same tmaining sel of The tink, while the pro-inired RITA muccl are intitally pre-tmine en other daas soumits (1z) Naniio
2. Bold for the best model (trained from scrvich) and box for the best pre pre-crined models. Running time comparison in Appendin C4
Table 3: ECG and human activity sensory classification tasks
Models
PTB-XI, (arrhythmia phenotype prediction)
HAR (huamnn action recognition)
Balanced Acc.
AUC-PR
AUROC
Balanced Acc.
Cohen's Kappa
Weighted
SPaRCNat [Tine al
20733
0.8275 0.0047
0.9040
0.0067
0.7550
0.0073
0.931
0.0160
0.9236 : 0.0199
0.9365
0.0155
ContraWi Taa a 2027
0.7532 0.0861
0.7549
0.0164
0,5258
00.190
0.906
0.0164
0.8879 1 000201
0.9055
0.0182
CNN- Trak-Tra Tren or. 2022
0.6650 0.0159
0.7175
0.0558
0.6996 1 0.0936
0.8690
0.0839
0.8273 1 0.0953
0.8352
1 0.1166
FFCL [Li al. 2022
0.7034
0.0522
0.7088
0.0053
0.5127 2: 0.0051
0.8519
0.0148
0.8216
: 0.0177
08508
01.038
ST-Truhstormer
Song
20213
0.7238
0.083
0.7775
0.0153
0.6003
0.0179
0.9336
0.0063
0.9213
1 0.0066
0.9337
0.0038
(Vanilla) BT07
0,8315 0.0008
0.0020
0.7493
0.9351 0.0160
0.9458
Prounined BIOT (Cardiolegy 6)
0,8350
0.0073
0.9128 1. 0,0094
0.7671 1 0.0166
Pretrained BIOT (Cardiology-12)
0.8421
0.0030
0.9221 a: 0,0075
0,7659 =: 0.0076
Bold for the best model, All modils use the simp unining sot of the taok, The Preirainal BLOP (Cmllology-5) 6) and Protrained 5107 (Candiolgy-- 12) are
pre tranned on Cardiology data (see Secioo and they da not apply to HAR data (due to different biosignal sypes).
7

---

-0- SPaRClet
-Q- ContraiRR
- CMN Tranddrberen
-0-
FFC.
-0- STTaadtommr
- (Vanlla) 8OT
-0- Protrained BOT (PREST-SHIS)
TUEV
TUEV
TUEV
0.55
0.55
0.50
0.50
0.74
0.45
0.70
0.40
0.25
0.35
0.30
0.64
Seizure
0.58
0.500
0.58
0.58
0.56
Dalnccd
0.54
0.52
0.425
0.52
0.50
0.100
0.375
0.46
0.350
No Mixinng
Mossing Segnnents
Missing Chamnnis
Masinn
h0 Mevinng
Missing Segenenis
Missing
Mlosing Both
No Mixinn
Mosing Segmentss
Missing
Misling
Soth
Charmels
Figure 3: Supervised learning with missing channels segments (on TUEV and III Seizure)
3.3
Setting (2) learning with missing channels and segments
The section simulates the TUEV dataset (16 channels and 5s per sample) and IIIC Seizure (16 channels and 10s per sample) to mimic the setting of supervised learning with missing channels and segments, which shows the strong performance of BIOT. We consider three missing cases:
Missing segments: Randomly mask out a segments (each segment spans for 0.5 seconds), a 2, 3, 4,5 with equal probability. The segment masking is applied separately for each channel. Missing channels: Randomly mask out channels, b 1, 2, 3, 4 with equal probability. We assume that the masking will not alter the underlying labels (the same assumption for other cases). Missing both channels and segments: Combine both masking strategies simultaneously.
To enable the baseline models compatible with the setting, use all zeros to impute the masked regions. The comparison is plotted in Figure 3 which shows that (i) all models decrease the performance with more missings while BIOT and the pre-trained BIOT less impacted (especially on Kappa and Weighted F1); (ii) "Missing channels" affects the performance more than "Missing segments", which makes as segment masking still preserves information from all channels.
3.4 Setting (3) unsupervised pre-training
This section shows that BIOT enables unsupervised pre-training on existing datasets with various formats. Note that, all the pre-trained models in this paper have similar scale (~3.3 million parameters). We defer the study of "scaling effect" of pre-trained BIOT to future work.
Pre-trained (PREST): This model is pre-trained on 5 million resting EEG samples (PREST) with 2,048 the batch size. We save the pre-trained model at the 100-th epoch.
Pre-trained (PREST+SHHS) This model is jointly pre-trained on 5M PREST and 5M SHHS EEG samples. Though two datasets have different sample formats, our model is able to use them together. Also, we use 2048 as the batch size and save model the 100-th epoch.
Pre-trained (Cardiology-12) is jointly pre-trained on raw data five datasets in Cardiology corpus (details in Appendix B.I. We use 1024 batch size and save model at the 100-th epoch. Pre-trained (Cardiology-6) is pre-trained similarly Pre-trained (Cardiology- 12), while we only utilize the first ECG leads. By contast, Pre-trained (Cardiology-12) uses full 12 leads.
We fine-tune the first two pre-trained EEG models on four EEG tasks and append the results to Table (also in Appendix C.T We fine-tune the last two pre-trained ECG models on PTB-XL. datasets in Table 3 The pre-trained models can be seamlessly applied to various downstream tasks with different sample formats. Results show that the pre-trained models can improve the performance consistently after fine-tuning the training set of downstream tasks in Table 2 and Table 3

---

3.5 Setting superviise pre-training on other tasks
This section shows that BIOT allows knowledge transfer from one task to another similar task with different sample formats. We pre-train on the training set of CHB-MIT, IIIC Seizure, TUAB and fine-tunes on TUEV (which has 16 channels and 5s duration). All datasets use 200Hz sampling rate. We design three sets of configurations for the pre-trained datasets: Format (i) uses the first 8 channels and 10s duration; Format (ii) uses the full 16 channels but only the first 5s recording; Format (iii) uses full 16 channels and full 10s recording. During fine-tuning, we then remove the prediction layers from these pre-trained model and add a new prediction layer to fit the TUEV dataset.
0.3338
0.10
Rafemrcee
ECC Selour toup)
TUAD lsus)
References
CH2- AIT (up!
IKC Socure bup!
TUAB (up!
0.735
0.7595
0.750
Vanilla BIOT
0.725
trainran BIOT (Format (i): channels, 10 seconds)
Wegine
0.700
Pre-trained BIOT (Format (il): 16 channels. seconds)
Pre-trained (Format (iii:: chennels, seconds!
0.25
0.50
References
Selrin
TUAR
Figure 4: Fine-tuned on TUEV from different supervised pre-trained models (best number in bold). Similar supervised fine-tuning analysis on CHB-MIT dataset shown in Appendix C.2
The fine-tuning results on TUEV are shown in Figure 4 where we also add the vanilla BIOT for reference. We find that (i) the model pre-trained on IIIC Seizure and TUAB are generally beneficial for the event classification task TUEV. The reason might be that TUAB and TUEV are both recorded from Temple University and share some common information, while IIIC seizure and TUEV both related to seizure detection and may share some latent patterns. (ii) More pre-training data leads to better results in the downstream task: though the pre-training configuration (16 channels, 5 seconds) aligns better with the TUEV data formats, the results show that configuration of (16 channels, 10 seconds) encodes longer duration and works consistently better. (iii) Compared to the TUEV results in Appendix C.I we also find that oftentimes the supervised pre-training (e.g., on IIIC seizure or TUAB) can be more effective than unsupervised pre-training (e.g., on SHHS and PREST).
3.6 Pre-trained on all EEG datasets
In this section, show that BIOT can leverage all six EEG resources considered in the paper. We obtain a Pre-trained (6 EEG datasets) model by loading the Pre-trained (PREST+SHHS) model and further train it on the training sets of CHB-MIT, IIC Seizure, TUAB, and TUEV. We add separate classification layers for four tasks. Essentially, this model is pre-trained on all six EEG datasets. To use the model, we still fine-tune on the training of downstream tasks and append the results to Table and Appendix C.I Apparently, Pre-trained (6 EEG datasets) outperforms the vanilla BIOT and is better than the unsupervised and the supervised pre-trained BIOT models in most cases.
4 Conclusions and Discussions
This paper proposes new biosignal transformer model (BIOT) that learns embeddings for biosignals with various formats. BIOT can enable effective knowledge transfer across different data and allow joint training on multiple sources. We conduct extensive evaluations on nine biosignal datasets and show that our BIOT is flexible and effective in various cross-data learning settings. Future efforts can explore the different types of biosignals and pre-training an all-in-one unified biosignal model. We hope our work can inspire more follow-up researches of large foundational models for biosignals.
Acknowledgments
This work was supported by NSF award SCH-2205289, SCH-2014438, and IIS-2034479. This project has been funded by the Jump ARCHES endowment through the Health Care Engineering Systems Center.

---

References
Alday, E. A. P., Gu, A., Shah. A. J., Robichaux, C., Wong, A.-K. I., Liu, C., Liu, F. Rad, A. B., Elola, A., Seyedi, S., et al. (2020). Classification of 12-lead eegs: the physionet/computing in cardiology challenge 2020. Physiological measurement, 41(12):124003.
Almutairi, H., Hassan, G. M., and Datta, A. (2021). Detection of obstructive sleep apnoea by eeg signals using deep learning architectures. In 2020 28th European signal processing conference (EUSIPCO), pages 1382-1386 1386. IEEE.
Anguita, D., Ghio, A., Oneto, L., Parra, X., Reyes-Ortiz, J. L., et al. (2013), A public domain dataset for human activity recognition using smartphones. In Esann, volume 3, page
Ba, J. L., Kiros, J. R., and Hinton, G. E. (2016). Layer normalization. arXiv preprint arXiv:1607.064500
Bahador, N., Jokelainen, J., Mustola, S., and Kortelainen, J. (2021). Reconstruction of missing channel in electroencephalogran using spatiotemporal correlation-based averaging. Journal of Neural Engineering, 18555505
Biswal, S., Sun, H., Goparaju, B., Westover, M. B., Sun, J., and Bianchi, M. T. (2018). Expert-level sleep scoring with deep neural networks. Journal of the American Medical Informatics Association, 25(12):1643-1650
Chen, T., Kornblith, S., Norouzi, M., and Hinton, G. (2020). simple framework for contrastive learning of visual representations. In International conference on machine learning, pages 1597- 1607. PMLR.
Clevert, D.-A., Unterthiner, T., and Hochreiter, S. (2015). Fast and accurate deep network learning by exponential linear units (clus). arXiv preprint arXiv:1511.07288
Cui, H., Liu, A., Zhang, X., Chen, X., Wang, K., and Chen, X. (2020). Eeg-based emotion recognition using an end-to-end regional-asymmmrrrcc convolutional neural network. Knowledge-Bord -aased Systems, 205:106243.
Dar, M. N., Akram, M. U., Khawaja, S. G., and Pujari, A. N. (2020). Cnn and lstm-based emotion charting using physiological signals. Sensors, 20(16):4551.
Devlin, J., Chang, M.-W., Lee, K., and Toutanova, K. (2018). Bert: Pre-training of deep bidirectional transformers for language understanding. arXiv preprint arXiv: 00.0000
Dosovitskiy, A., Beyer, L., Kolesnikov, A., Weissenborn, D., Zhai, X., Unterthiner, T., Dehghani, M., Minderer, M., Heigold, G., Gelly, S., et al. (2020). An image is worth 16x16 words: Transformers for image recognition scale. arXiv preprint arXiv:2010.11922
Du, Y., Xu, Y., Wang, X., Liu, L., and Ma, P. (2022). Eeg temporal-spatial transformer for person identification. Scientific Reports, 12(1):14378.
Ge, W., Jing, J., An, S., Herlopian, A., Ng, M., Struck, A. F., Appavu, B., Johnson, E. L., Osman, G., Haider, H. A., al. (2021). Deep active learning for interictal ictal injury continuum eeg patterns. Journal of neuroscience methods, 351:108966.
Gong, Y., Chung, Y.-.., and Glass, J. (2021). Ast: Audio spectrogram transformer. arXiv preprint arXiv:2104.0177.
Greco, A., Valenza, G., Lázaro, J., Garzón-Rey MarzónnRRey.a M., Aguiló, J., De-la Camara, C., Bailón, R., and Scilingo, E. P. (2021). Acute stress state classification based on electrodermal activity modeling. IEEE Transactions on Affective Computing.
Grill, J.-B., Strub, F., Altché, F., Tallec, C.,, Richemond, P., Buchatskaya, E., Doersch, C., Avila Pires, B., Guo, Z.., Gheshlaghi Azar, M., et al. (2020). Bootstrap your own latent-a new approach to self-supervised learning. Advances in neural information processing systems, 12222244..
10

---

Harati, A., Golmohammadi, M., Lopez. S., Obeid, I., and Picone, J. (2015). Improved eeg event classification using differential energy. In 2015 IEEE Signal Processing in Medicine and Biology Symposium (SPMB), pages 1-4. IEEE.
He, K., Fan, H., Wu, Y., Xie, S., and Girshick, R. (2020). Momentum contrast for unsupervised visual representation learning. In Proceedings of the IEEE/CVE conference on computer vision and pattern recognition, pages 9729-9738.
He, K., Zhang, X., Ren, S., and Sun, J. (2016). Deep residual learning for image recognition. In Proceedings of the IEEE conference computer vision and pattern recognition, pages 770-778.
Isin, A. and Ozdalili, (2017). Cardiac arrhythmia detection using deep learning. Procedia computer science, 120:268-2177
Jing, J., Ge, W., Hong, S., Fermandes, M. B., Lin, Z., Yang, C., An, S., Struck, A. F., Herlopian, A., Karakis, I., et al. (2023). Development of expert-level classification of seizures and rhythmic and periodie patterns during eeg interpretation. Neurology.
Jing, J., Sun, H., Kim, A., Herlopian, A., Karakis, I., Ng. M., Halford, J., Maus, D., Chan, F., Dolatshahi, M., et al. (2020). Development of expert-level automated detection of epileptiform discharges during electroencephalogram interpretation. JAMA neurology, 77(1):103-108.
Katharopoulos, A., Vyas, A., Pappas, N., and Fleuret, F. (2020). Transformers are rnns: Fast autoregressive transformers with linear attention. In Proceedings of the International Conference on Machine Learning (ICML).
Kim, M.-G,, Ko, H., and Pan, S. B. (2020). A study user recognition using 2d ecg based on ensemble of deep convolutional neural networks. Journal of Ambient Intelligence and Humanized Computing, 11:1859-18 1867.
Klem, G. H., Lüders, H., Jasper, H. H., and Elger, C. the international federation. the international federation of clinical neurophysiology. Electroen- cephalography and clinical neurophysiology. Supplement, 52:3-6.
E. (1999). The ten-twenty electrode system of
Kostas, D., Auele-Oelleete,, S., and Rudzicz, F. (2021). Bendr: using transformers and a contrastive self-supervised learning task to learn from massive amounts of eeg data. Frontiers in Human Neuroscience, 15:653659.
Lawhern, V. J., Solon, A. J., Waytowich, N. R., Gordon, S. M., Hung, C. P., and Lance, J. (2018). Eegnet: a compact convolutional neural network for eeg-based brain-computer interfaces. Journal of neural engineering, 15550035
Li, H., Ding, M., Zhang, R., and Xiu, C. (2022). Motor imagery eeg classification algorithm based on cnn-lstm feature fusion network. Biomedical signal processing and control, 72:103342.
Lin, T.Y., Goyal, P., Girshick, R., He, K., and Dollár, P. (2017). Focal loss for dense object detection. In Proceedings of the IEEE international conference on computer vision, pages 2980-2988.
I Liu, J., Zhang, I.., Wu, H., and Zhao, H. (2021). Transformers for eeg emotion recognition. arXiv preprint arXiv:2110.06. 06553..
Liu, Y., Ou, M., Goyal, N., Du, J., Joshi, M., Chen, D., Levy, O., Lewis, M., Zettlemoyer, L., and Stoyanov, V. (2019). Roberta: A robustly optimized bert pretraining approach. arXiv preprint arXiv:190..1192.
Lopez, S., Suarez, G., Jungreis. D., Obeid, I., and Picone, J. (2015). Automated identification of abnormal adult eegs. 2015 IEEE Signal Processing in Medicine and Biology Symposium (SPMB), pages 1-5. IEEE.
Nagabushanam, P., George, S. T., Davu, P., Bincy, P., Naidu, M., and Radha, S. (2020). Artifact removal using elliptic filter and classification using Id-enn for eeg signals. In 2020 6th International Conference on Advanced Computing and Communication Systems (ICACSS), pages 551-556. IEEE.
11

---

Nyquist, H. (1928). Certain topics in telegraph transmission theory. Transactions of the American Institute of Electrical Engineers, 4772)4617-644
OpenAl (2023). Gpt-4 technical report.
Parvaneh, S., Rubin, J., Babaeizadeh, S., and Xu- Ku-Wilson, M. (2019). Cardiac arrhythmia detection using deep learning: A review. Journal of electrocardioloyy. 57:S70-S74,
Peh, W. Y., Yao, Y., and Dauwels, J. (2022). Transformer convolutional neural networks for automated artifact detection scalp eeg. In 2022 44th Annual International Conference of the IEEE Engineering in Medicine & Biology Society (EMBC), pages 3599-3602. IEEE.
Phan, H. and Mikkelsen, K. (2022). Automatic sleep staging of eeg signals: recent development, challenges, and future directions. Physiological Measurement.
Quan, S. F., Howard, B. V., Iber, C., Kiley, J. P., Nieto, F. J., O'Connor, G. T., Rapoport, D. M., Redline, S., Robbins, J., Samet, J. M., et al. (1997). The sleep heart health study: design, rationale, and methods. Sleep. 20 10(12):1077-1885.
Sakhavi, S., Guan, C., and Yan, S. (2018). Learning temporal information for brain-computer interface using convolutional neural networks. IEEE transactions on neural networks and learning systems, 29(1 5569-562..
Schirmmesster, R. T., Springenberg, J. T., Fiederer, L. D. J., Glasstetter, M., Eggensperger, K., Tangermann, M., Hutter, F., Burgard, W., and Ball, T. (2017). Deep learning with convolutional neural networks for eeg decoding and visualization. Human brain mapping, 38(1 (((1)):53115422)
Shannon, C. E. (1949). Communication in the presence of noise. Proceedings of the IRE, 37(1)100-21.
Shoeb, A. H. (2009). Application of machine learning to epileptic seizure onset detection and treatment. PhD thesis, Massachusetts Institute Technology.
Song, Y., Jia, X., Yang. L., and Xie, L. (2021). Transformer-based spatial-tempont feature learning for ceg decoding, arXiv preprint arXiv:2106.11170.
Srivastava, N., Hinton, G., Krizhevsky, A., Sutskever, I., and Salakhutdinov, R. (2014). Dropout: a simple way to prevent neural networks from overfitting. The journal of machine learning research, 15(1):1929-1958.
Suhaimi, N. S., Mountstephens, J., Teo, J., et al. (2020), Eeg-based emotion recognition: A state-of- the-art review of current trends and opportunities. Computational intelligence and neuroscience, 2020.
Vaswani, A., Shazeer, N., Parmar, N., Uszkoreit, J., Jones, I.., Gomez, A. N., Kaiser, E.., and Polosukhin, I. (2017). Attention is all you need. Advances in neural information processing systems, 30.
Venkatachalam, K., Devipriya, A., Maniraj, J., Sivaram, M., Ambikapathy, A., and Iraj. S. A. (2020). novel method of motor imagery classification using eeg signal. Artificial intelligence medicine. 103:10
Wagner, P., Strodthoff,, N., Bousseljot, R.-D., Kreiseler, D., Lunze, F. Samek, W., and Schaeffter, (2020). Ptb-xl, a large publicly available electrocardiograply dataset. Scientific data, 7(1):154.
Wang, S., Li, B. 7.,, Khabsa, M., Fang, H., and Ma, H. (2020). Linformer: Self-attention with linear complexity. arXiv preprint arXiv:2006.0477 04768.
Yang. C., Qian, C., Singh, N., Xiao, C. D., Westover, M., Solomonik, E., and Sun. J. (2022). Atd: Augmenting cp tensor decomposition by self supervision. Advances in Neural Information Processing Systems, 35:32039-3200
Yang, C., Westover, M. B., and Sun, (2023). Manydg: Many-domain generalization for healthcare applications. In The Eleventh International Conference Learning Representations.
12

---

Yang, C., Xiao, D., Westover, M. B., and Sun, J. (2021). Self-supervised eeg representation learning for automatic sleep staging. arXiv preprint arXiv:2110.15278.
Zhang, G.-Q., Cui, L., Mueller, R., Tao, S., Kim, M., Rueschman, M., Mariani, S., Mobley, D., and Redline, S. (2018). The national sleep research resource: towards a sleep data commons. Journal of the American Medical Informatics Association, 25(10):1351-335;
Zhang, R., Zong, Q. Dou, L.,, and Zhao, X. (2019). A novel hybrid deep learning scheme for four-class motor imagery classification. Journal of neural engineering, 16(6):066004.
Zhang. X., Zhao, Z., Tsiligkaridis, T., and Zitnik, M. (2022). Self-supervised contrastive pre-training for time series via time-frequene consistency. arXiv preprint arXiv:2206.08423 08496..
Zhang, Y., Chen, J., Tan, J. H., Chen, Y., Chen, Y., Li, D., Yang, L.. Su. J., Huang, X., and Che, W. (2020). An investigation of deep learning models for eeg-based emotion recognition. Frontiers in Neuroscience, 14:622759.
13

---

A Notaion Table
The listed the notations used in the main text in the following Table
Table Notations used in BIOT
Symbols
Descriptions
Te N
the number of channels in biosignal, such 16
J R E R
the longth of biosignals, such as 10s x 200/s (use complete sample for case notation)
mulit-channel biosignal
S/a R-
i-th channel of the biosignal.
It+
sampling rate, such
3+
the longth of the biosignal token, such 1s 200/fs
PeR!
the overlap length between two neighboring tokens, such as 0.5a x 2008s. ( p
KENT
k-th token the 1-th channel denoted by Sty (t p)(k - 1) p)(k 1) + t) which has length number of tokens in channel
K KEN!
NEX
number of tokens in tokenized biosignal "sentence"
4 MEN
the dimension of the token embedding
X K
the tokenized biosignal "sentence"
bEN
new dimensions of tokens after self-atteation module key,
W W w pioxt
query matrices self attention medule low-rank projection matrices
E, R
de N+
the reduced rank for self-attention matrices, d < N the output of the self-attention module
H 2x
$ e t 7.% T
the perturbed biosignal sample
the real and predicted emberklings of temperature hyperparameter the contrastive loss the identity matrix in the contrastive loss, has the size of (batch size, batch size)
6 (0, 4
the discrete uniformly distributed variable for the number of masked segments in Section 3.3 discrete uniformly distributed variahle for the mumber of masked channels in Section 3.5
be 2,
B Details of Datasets and Experimental Settings
B.1 More for Datasets and Processings
We provide more descriptions on each dataset in this section.
For EEG datasets. First, the 16 derivations (in 10-20 international system) are "FP1-F7", "F7-T7", "T7-P7", "P7-O1", "FP2-F8"", "F8-T8", "T8-P8", "P8-O2", "FP1-F3", "F3-C3", "P--O",, "FP2-F4", "F4-C4", "C4-P4", " "P4-02".
Sleep Heart Health Study (SHHS) (Zhang et al., 2018 Quan et al. 1997) is a multi-center
cohort study from the National Heart Lung & Blood Institute assembled study sleep-disordered breathing, which contains 5,445 recordings. The data is accessible upon request in their website Each recording has 14 Polysomnography (PSG) channels, and the recording frequency is 125.0 Hz. We use the C3/A2 and C4/A1 ERG channels. The dataset is released with sleep annotations. We use the existing codes and split each recordings into 30-second samples. In this study, we use SHHS samples for unsupervised pre-training without the original labels.
PREST is a private dataset recorded in hospital sleep lab, primarily for seizure and abnormal EEG detection purpose (such spikes). The local IRB waived the requirement for informed consent for this retrospective analysis of EEG data. We follow the clinician's 's instructions and split each recordings into 10 seconds without labels. the experiment, we use for EEG model pre-training.
The CHB-MIT database Shoeb 2009 publicly available, which is collected the Children's Hospital Boston, consists of EEG recordings from pediatric subjects with intractable seizures. The dataset is under Open Data Commons Attribution License vl.0 and is used to predict whether the EEG recordings contain seizure signals. Each recording initially contains 23 bipolar channels and
"atts://sccpdaat..o//lttttsssss
14

---

we select the 16 standard derivations in the experiments. We utilize the existing preprocessing and follow the typical practices further split each recordings into 10-second non-overlapping samples by default. Since the dataset is highly imbalanced, we use 5 seconds overlaps to split the seizure regions (which could potentially double the positive samples), After processing, the positive ratio in the training set is around 0.6%.
IIC Seizure is requested from Ge et al. (202]); Jing et al. (023), and we follow the license and
usage statements in Jing et al. (2023). The samples follow 16 derivations and span 10-second
signals at 200Hz. This dataset is used for predicting one of the six classes: lateralized periodic
discharges (LPD), generalized periodic discharges (GPD), lateralized rhythmic delta activity
(LRDA), generalized rhythmic delta activity (GRDA), Seizure types, and Other.
TUH Abnormal EEG Corpus (TUAB) (Lopez et al. 2015) and TUH EEG Events (TUEV) (Harati et al.
2015) is accessible upon request Temple University Eleetroencephalography (EEG)
Resources We process both datasets to follow the 16 EEG derivations.
For ECG datasets. We use the Cardiology collection to pre-train the ECG models and apply on downstream supervisd PTB-XL task.
The Cardiology collection (Alday et al. 2020) is publicly available at physionet which was used in the PhysioNet/Computtig in Cardiology Challenge 2020. This collection is under Creative Commons Attribution 4.0 International Public License g In this study, we use five sets from the training portion of the collection (It has in total six sets. Another one overlaps with the PTB-XI dataset, and thus we drop it for pre-training), which contains recordings from CPSC2018 8 (6,877 recordings), CPSC201 data, 3,453 recordings), St Petersburg Incart (12-lead Arrhythmia Database, 74 recordings), PTB (Diagnostic ECG Database, 516 recordings). Georgia (12-Lead ECG Challenge Database, 10,344 recordings). For preprocessing, we extract 10-second samples from each recording with 0.5s as the overlapping window (for obtaining more unsupervised trianing corpus). All the samples merged together as an unsupervised pre-training ECG corpus of nearly 0.5 million samples. We pre-train a Pre-trained BIOT (Cardiology-12) on all the channels and a Pre-trained BIOT (Cardiology-6) on the first 6-channels of all samples. The sample sizes are different from the below PTB-XL dataset.
(China 12-Lead ECG Challenge Database unused CPSC 2018
Physikalisch-Tecchishhe Technische Bundesanstalt (PTB-XL) 10 Wagner et al. 2020) is a publicly available large dataset of 12-lead ECGs from 18885 patients. It is under the Creative Commons Attribution 4.0 International Public License !1 The raw waveform data was annotated by up to two cardiologists, who assigned potentially multiple ECG statements to each record up to 27 diagnoses: 1:1st degree AV block, 2:Atrial fibrillation, 3:Atrial flutter, 4:Bradycardia, 5:Complete right bundle branch block, 6:Incomplete right bundle branch block, 7:Left anterior fascicular block, &: axis deviation, 9:Left bundle branch block, 10:Low QRS voltages, 1 11:oonpeciific intraventricular conduction disorder, 12:Pacing rhythm, 13:Premature atrial contraction, 14:Premature ventricular contractions, 15:Prolonged PR interval, 16:Prolonged QT interval, 17:Q wave abnormal, 18:Right axis deviation, 19:Right bundle branch block, 20:Sinus arrhythmia, 21:Sinus bradycardia, 22:Sinus rhythm, 23:Sinus tachycardia, 24:Supravenrricular premature beats, 25:T wave abnormal, 26:T wave inversion, 27: Ventricula premature beats. We following clinical knowledges and further groups them into six broader categories: Arrhythmias, Bundle branch blocks and fascicular blocks, Axis deviations, Conduction delays, Wave abnormalities, Miscellaneous. Each recordings can be associated to multiple categories. In this paper, we conduct the "Arrhythmias" phenotyping prediction task. If the recordings have at least one diagnosis belonging to the Arrhythmias group, then label them positive, otherwise negative.
For human activity sensory data. Human activity recognition (HAR) dataset 2013) publicly available UCI machine learning repository. The data is collected from smartphone accelerometer and gyroscope data with 3D coordinates to detect six actions: walking, walking
12 Anguita et al.
7 http://ppppppooppppsssoc/ooppppp
1.0.//
15

---

upstairs, walking downstairs, sitting, standing, laying. The samples are already splitted and provided in the original datasets.
B.2 More for Experimental Settings
For model implementation, the SPaRCNet code is requested from the authors (Jing et al. 2023).
the Contra WRR code is downloaded and modified upon the github 13 CNN-Transformer is easily
implemented following the Fig. of the original paper (Peh al. 2022) FFCL (Li et 2022
combines CNN model and a LSTM model for learning separete representations and then merges them before the final prediction layer, the implementation of TTaarosffrrrrr refer to this repo
The imear-complexit attention module is referred to this repo T5 in our BIOT implementation.
For all EEG tasks, resample the datasets into 200Hz. The ECG tasks use 500Hz, and the HAR tasks use 50Hz by default. For each specific tasks, we have to adjust the baseline model architectures (e.g, number of layers, input channel sizes, etc) accordingly since the input data have various formats. While for our BIOT, we only adjust the fft size based on their sampling rate (200Hz for EEG, 1000Hz for ECG, 100Hz for HAR) and use 0.5s, 0.2s, and 0.1s as the hop length (i.e., overlaps) in three signal types. These model configurations are chosen by testing several combinations based on the validation performance and we select the best one. For our BIOT model, we use 8 as the number of head, as the number of transformer layers, and T = 2 as the temperature in unsupervised pre-training by default. We use the Adam optimizer with learning rate 1 10- and 1 x 10 as the coefficient for 1.2 regularization by default. We use the pytorch lightning framework (with 100 the max epoch) to handle the training, validation, and test pipeline by setting AUROC as the monitoring metire for binary classification and Coken's 's Kappa as the monitoring metric for multi-class classification in the validation. More details can refer to our Supplementary codes. Below, we provide the definition of each metric used in the paper.
Balanced Accuracy defined the average of recall obtained on each class. It is used for both binary classification and multi-class classification.
AUC-PR is the area under the precision recall (PR) curve for binary classification task.
AUROC is the area under the ROC curve, summarizing the ROC curve into an single number that describes the performance of model for multiple thresholds at the same time. It is used for binary classification.
Coken's Kappa is a statistic that measures inter-annotator agreement, which is usually used for imbalanced multi-class classification task. The calculation can refer to sklearn metrics 16
Weighted F1 is used for multi-class classification in this paper, which is a weighted average of individual F1-scoree from each class, with each score weighted by the number of samples in the corresponding class.
C Additional Results
This section provides additional experimental results to support claims in the main paper.
C.1 Additional Experiments on TUEV and TUAB
We have provided the supervised learning results on EEG dataset IIC Seizure and CHB-MIT in the main text. For completeness, we provide similar comparison results on TUAB and TUEV below in T which show a similar trend that our BIOT shows better performance against baseline models, and the pre-trained BIOT models can bring significant improvements on two downstream tasks, especially on TUEV. For TUEV, we also append the results of all different pre-trained models (e.g., train from scratch, supervised training, unsupervised training, etc) the end in Table
"hhtp:://ithubbbc "baaps//ginuh..m//uuuiraiiinnni wwwp://eeeiiiii/////aaaaaatttttn 1 http:///we..oooeeooooooooo//oeee
kapppriescccceppppppppppppp kappa_score.hhm html
16

---

Table 5: Additional Supervised Learning Results on TUAB
TUAB (abnormal detection)
Models
Balanced Acc.
AUC-PR
AUROC
SPaRCNet
0.7896 0.0018
0,8414 = 0.0018
0.8676 + 0.0012
ContraWR
0.7746 0.0041
0.8421 + 0..0104
0.8456 1: 0.0074
CNN- TeN-Tranffrmerr
0.7777 0.0022
0.8433 = 0.0039
0.8461 = 0.0003
FFCL
0.7848 0.0038
0.8448 = 0.0065
0,8569 1 0.0051
ST-Trasfformrr
0.7966 0.0023
0.8521 1: 0.0026
0.8707 1 0.0019
(Vanilla) BIOT
0.7925 0.0035
0.8707 0.0087
0.8691 + 0.0033
Pro-trained BIOT (PREST)
0,7907 0,0050
0.8752
0.0051
0.8730 + 0.0021
Pre-trained BIOT (PREST+SHHS)
0,8019 0.0021
0.8749 0.0054
0,8739 + 0.0019
Pre-trained BIOT (6 EEG datasets)
0.7959 0.0057
0.8792 0.0023
0,8815 0.0043
Bold for the best model (trained from scratch) and box for the best pre-trained models.
Table 6: Additional Supervised Learning Results on TUEV (All-in-one-table comparison)
TUEV (event type classification)
Models
Balanced Acc.
Coken's 's Kappa
Weighted F1
(Training from scratch in Section 3.2
SPaRCNet
0.4161 0.0262
0.4233 : 0.0181
0.7024
0.0104
ContraWR
0.4384 = 0.0349
0.3912 = 0.0237
0,6893 + 0,0136
CoN-Transformer
0,4087 = 0.0161
0.3815 = 0.0134
0.6854 0.0293
FFCL
0.3979 0.0104
0,3732 0.0188
0,6783 0,0120
ST.Transformer
0.3984 + 0.0228
0.3765 + 0.0306
0.6823 + 0.0190
(Vanilla) BIOT
0,4682 = 0.0125
0.4482
# 0.0285
0.7085 + 0.0184
(Unsupervised pre-trained models Section
Pre-trained BIOT (PREST) Pre-truined BIOT (PREST+SHHS)
0.5207
0.4932
0,0301
0.7381
0.0169
0.5149 = 0.0292
0,4841 = 0.0309
0,7322 + 0.019
(Supervised pre-trained models in Section 3.5
Pre traitraned BIOT (pre- trained on CHB HIT with 8 channels and 10s)
0.4123 :: 0.0087
0.4285 # 0.0065
0.6989 + 0.0015
Pre-tmained BIOT (pre-trained on CHB-MIT with chunnels and 5s)
0.4218 + 0.0117
0.4427
0.0093
0.7147 0.0058
Pre-trained BIOT (pre-trained on CHB-MIT with 16 channels and 10s)
0.4344 # 0.0065
0.4719 + 0.0231
0.7280 + 0.0126
Pre-trained BIOT (pre-trained on IIIC scizure with 8 channels and 10s)
0.4956 = 0.0552
0.4719 = 0.0475
0.0220
= 0.0220
Pre-truined BIOT (pre-trained on seizure with chunnels and 5s)
0.4894 + 0.0189
0,4881 + 0.0045
0.7348 0.0056
Pre-truined BIOT (pre-trained on IICC scizure with 16 channels and 10s)
0.4935 = 0.0288
0,5316 = 0.0176
0.7555 = 0.0111
Pre-truined HIT (pre-trained on TUAB with 8 channels and 10s)
0.4980 + 0.0384
0.487 + 0.0535
0.7044 + 0.0365
Pre-trained BIOT (pre-trained TUAB with channels and 5s)
0.4954 0.0305
0,5053
0.0079
0.7447 0.0049
Pre-trained BICT (pre-trained on TUAB with 16 channels and 10s)
0.5256 = 0.0348
0,5187 = 0.0160
0.7504 = 0.002
(Supervised unsupervised pre-trained model in Section 3.6
Pre-trained BIOT (6 EEG datasets)
0.5281 = 0.0225
0,5273 = 0.0249
0,7492 + 0.0082
C.2 Additional Experiments on CHB-MIT
This section performs a similar experiment on CHB-MIT, similar to Section 3.2 training set of III Seizure (which has 16 channels and 10s duration), TUAB (which has 16 channels and 10s duration), TUEV (which has 16 channels and 5s duration) and fine-tunes on CHB-MIT (which has 16 channels and 10s duration). All datasets use 200Hz sampling rate. We design five sets of configurations for the pre-trained datasets: Format (i) uses the first 8 channels and 10s duration; Format (ii) uses the full 16 channels but only the first 5s recording; Format (iii) uses full 16 channels and full 10s recording; Format (iv) uses 8 channels and 5s recording. and Format (v) uses full 16 channels and 2.5s recording. The last two are only for the TUEV dataset. During fine-tuning, we then remove the prediction layers from these pre-trained model and add new prediction layer to fit the CHB-MIT dataset.
We pre-train on the
The results are reported in Figure 5 which shows that the supervised pre-training on both IIC seizure and TUEV can help improve the downstream performance on CHB-MIT task compared to training from scratch. The reason is that IIIC Seizure is on multiple seizure type classification while CHB-MIT is on binary "seizure or not" classification, and the context of both tasks are fairly related. Although TUEV is not entirely on seizure related classification, some classes in TUEV are seizure subtypes (such as GPED, PLED), and thus its supervisd pre-trained models can also bring benefits for the CHB-MIT MIT task.
17

---

0.8885
6.6813
0.50
0.55
tal Ker
0.65
Colae's Kegpa
6.9
0.75
0.50
0.70
0.85
Rafermtce BCC Seluum (up))
TILK (wip)
TUFF (we))
IEC Sel.Se (upp
THAR (NAR)
TIPY (eap?
0.2639
Vanilla BIOT
We Mhoooe
6.25
Pre-trained BIOT (Format (0): channels, 10 seconds)
BIOT (Format
channels. seconds)
0.20
pre-trained BIOT (Format (il: 16 channels, seconds)
Pre trained BIOT (Format (v): 16 channels, 2.5 seconds)
Pre Pre-trained BIOT (Format [il];: 16 channels, 10 seconds)
6.10
Pre-trained BIOT
chennels, seconds)
Referettee
IBC Dup!
TIUM (eup)
(41))
Figure 5: Fine-tuned on CHB-MIT from different supervised pre-trained models. IIIC Seizure and TUAV datasets follow Format (i)ii)(iii, while TUEV follows the Format (iv)(v)(ii).
C.3 Ablation Studies on Hyperparameters
This section provides ablation studies on three hyperparameters in data processing: target sampling rate r, token length 4, and the overlap size p between two neighboring tokens. We use EEG datasets as example: IIIC Seizure and TUAB. The default configuration in the main paper is (1)
sampling: 200H2, 2, (2) token length: 1s r, (3) overlaps: p 0.5s r as reference.
also show an illustration for tokenization with overlap in Figure 6
1s
1s
1s
Teben
Token
Tolen3
3s (just show one channel)
1s
15
15
0.5
0.5
Figure 6: Illustration on tokenization with token length t and overlap p. The example shows one channel with 3 seconds. The configuration of (t r. 0) gives tokens while the configuration of (l r,p = 0.5r) gives 5 tokens. Different configurations lead to different lengths.
C.3.1 Ablation Study on Target Sampling Rate
In this experiment, we fix the coefficient "1" and "0.5" in (2)(3) and conduct ablation study on the target sampling rate r. The original IIC Scizure data is at 200Hz and the TUAB data is at 256Hz. For IIIC Seizure, vary the sampling rate 26Hz, 50Hz, 100Hz, 150Hz, and 200Hz. For TUAB, we vary the sampling rate to 50Hz, 100Hz, 150Hz, 200Hz, 250Hz, and 300Hz. We use the tool scipy.signan www.ol.sssmp.........mm.. The evaluations conducted under three different random seeds and the mean and standard deviation values are reported.
For IIIC Seizure, we can observe that a higher sampling rate could give slightly better performance, especially on balanced acc. and coken's kappa. The reason that higher sampling rate can preserve more detailed high-frequency biosignal information. The results on TUAB shows that the perfor- mances increase and then decrease slightly during increasing the sampling rate. We guess that with the increasing of initially the performance improves due obtaining more information. Later,
18

---

higher sampling rate does not bring more benefits but unnecessary frequency bands might incur some noise. We also conjecture that different tasks might have diverse sensitivity to the the frequency bands. For example, the task on IIIC seizure is to classify different seizure types, which may need to capture minor clues from high-frequency waves (such as Gamma waves (50-100Hz)) while the TUAB dataset is for abnormal detection, and using brain waves under 50Hz might be enough for the task. In sum, the target sampling rate should be selected based on the predicting targets.
IK Sebon
6.30
355
318
5372
690
6350
855
26
69
290
18
16
150
200
31115
09775
1.88
31325
3190
03130
3185
6.00
heanping
vpligg 1034
Figure 7: Ablation Study on Target Sampling Rate
C.3.2 Ablation Study on Token Lengths
In this experiments, we fix (1)(3) and conduct ablation study on the coefficient in token length f. Both datasets have 10s the entire sample length and 0.5s as the overlap lengths. For both of the datasets, we vary the token length coefficient to 0.75s, 1s, 1.5s, 2s, 2.5s, 5s. The evaluations are conducted under three different random seeds and the mean and standard deviation values are reported.
For each configuration, we also set the fft size to match the token length, which means that 5s token duration will bring more frequency information. However, we find that by increasing the token lengths, the model performance starts to decrease. Model performances on IIIC Seizure starts to decrease after is while the performance on TUAB decreases after 2s. The reason could be that longer token length (i.c.. frequency bands) do not provide extra benefits for learning the tasks; (ii) given the increasing token lengths t, the total biosignal "sentence" length, which is d decrease (here, J 10r is the channel duration, length). For example, with ( 5r as the token lengths, the number of tokens becomes 2 per channel while the number tokens per channel 19 the default configuration with r. The performance drops is due to transformer models will be less beneficial in shorter "sentence"s.
will the token length, p 0.5r is the overlapping
10-t
T-p
2-0.5
3.53
3.57
035
636
218
053
0.300
0.355
085
8.50
8.35
Figure 8: Ablation Study on Token Lengths /
19

---

C.3.3 Ablation Study on Overlapping Lengths
In this experiments, we fix (1)(2) and conduct ablation study on the coefficient in the overlap length p. Both datasets have 10s as the entire sample length and Is as the token lengths. For both of the datasets, we vary the overlap length coefficient to 0.875s, 0.75s, 0.5s, 0.25s, Os. The evaluations are conducted under three different random seeds and the mean and standard deviation values are reported.
Based on the "sentence" length formula / +1, smaller overlap length p will decrease the "sentence" length. On both datasets, we find that larger overlap can brings better results due that the biosignal "sentence" becomes longer, and Transformer models can be more beneficial in the cases. Another reason is that with larger overlaps, neighboring tokens can capture more transitioning information and help the transformer model to better capture the temporal dynamics.
t-p
c90
8.4
5912
8.33
095
y Lirgha
Contogrn LengPap lo (1011
Longle 1E
31375
T148
0.22
08:22
31900
31822
M00
8.55
08513
0827c
05:
135
1.51
Figure 9: Ablation Study on Overlapping Lengths p Between Tokens
C.4 Running Time Comparison
This section compares the running time of all models on all supervised learning tasks (CHB-MIT, TUAB, IIIC Seizure, TUEV, PTB-XI XL. HAR). When recording the running time, duplicated the environment mentioned Section 3.2 stopped other programs, and ran the models one by one on one GPU. We record the first 10 epochs of all models and report the per epoch mean and standard deviation of the time cost in Table 7 In the setting, we use 512 as the batch size for CHB-MIT, TUAB, TUEV, and PTB-XI XL, 256 batch size for IIIC Seizure, and 64 as batch size for HAR.
The results show that our model has similar running time profile as CNN- CNN-Transformer and the ST-Transformer models. These two baselines and our BIOT are all transformer based models and have the same number of heads and layers. BIOT uses linear self-attention module which should be faster than CNN-Transformer and the ST-Transformer, however, our sequence structure are also generally longer due to the flattening transformation. We also find that the CNN models (SPaRCNet and ContraWR) are faster than the transformer based models in our experiments.
In all, the running time of all models are in the same magnitude, and the actual running time on certain applications can vary due to the signal length, number of channels, selected number of CNN or transformer layers or attention heads. The reported running time comparisons here work as references using our selected model architectures. We think the running time of BIOT is acceptable given its decent performance.
Table 7: Running time comparison (seconds per epoch)
Model
CHB-MIT
TUAB
IIC Scizure
TUEV
HAR
PTB-XL
SPaRCNot
32.4417
17.4635
1.0861
24.8728
2.339
1,9485
7.3850 0.0599
ContraWR
24.7308 = 0.7905
13,4650 + 0.4554
14.9499
0.9233
5.3357 :: 0.2003
2.3255 = 0.1308
5.1978 + 0.0366
CNN- Transforfform
53.3355
18880
25.8983
1 0.7969
25.7742
1.299
9.0415
1: 0.2501
3,0207 2 0.0433
7.9851 1 0.0400
FFCL
43.6103 = 0.7927
21.6868
0.5668
23,1682
1.0757
7.0863
-: 0.1333
2.6689
0.0973
6.513 0.0228
ST. TT-raasffommrr
50.4725 = 0.9978
24.5641
4 0.4770
26.3151
3.3320
7.9027
1: 0.0758
2.7954
1 0.06998
11.1495
1: 0.0200
(Vanilla) BIOT
55,5780
0.4229
25.2788
0.1200
25.4500
4.083S
8.1812
1: 0.1228
2,9560 0.0563
12.1791 0.0452
20

---

C,5 Discussion on handling long recordings and multiple channels
Currently, the BIOT model already have two designs to handle long recordings and more channels: (i) the current model uses linear complexity transformer, so that the complexity scales linearly with sample lengths and channel sizes; (ii) we can remove the token overlaps and enlarge the token sizes to reduce the token numbers (i.e., "sentence" length). With minor adjustments, the BIOT model can better handle long recordings and multiple channels. For long recordings, we could segment the recordings into 10-30s samples and then apply our BIOT on each sample and finally use top level LSTM or Transformer learn sequence embedding. For multiple channels (more than 256 channels), we can group neighboring channels or symmetric channels and tokenize them together, which could greatly shrink the final "sentence" length. However, additional adjustments are not needed this paper and further discussion is beyond the scope of this paper.
21


Reviews:
[{"review_id": "hzAi2MtlOM", "reviewer": "Reviewer_ve81", "summary": "The paper presents a Biosignal Transformer (BIOT) model that can be pre-trained from multiple data sources and fine-tuned on different downstream biosignal tasks. The model tokenizes diverse biosignals into unified “biosignal sentences” and adds channel embeddings and relative position embeddings to preserve spatio-temporal features. The BIOT model is versatile and applicable to various biosignal learning settings across different datasets. Comprehensive evaluations on EEG, ECG, and human activity sensory signals demonstrate that BIOT outperforms robust baselines in common settings and facilitates learning across multiple datasets with different formats.\n\n", "questions": "Why were these specific datasets chosen for evaluation?\nHow would the proposed model perform on large datasets like MIMIC-IV?\nHow would the proposed model compare to reference models like SimCLR or MAEs?\n", "limitations": "The most significant limitation of this paper is the experimental setup. The selected baselines do not include some reference models like SimCLR or MAEs. There are no scaling experiments which seems to be the most important finding in GPT-style models. If we want to create the equivalent of a GPT for biosignals, we need to source and combine probably hundreds of similar datasets and think hard about the proportion of each modality (EEG, movement, ECG etc). However, the authors left this for future work.\n\n", "rating": 7, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "The paper presents a novel approach to handling diverse biosignals by tokenizing them into unified “biosignal sentences”. The proposed BIOT model is versatile and applicable to various biosignal learning settings across different datasets. The comprehensive evaluations on EEG, ECG, and human activity sensory signals demonstrate the robustness of the proposed model.\n\n", "weaknesses": "The paper does not provide motivation for why the authors picked these specific datasets. There is no discussion about large datasets like MIMIC-IV that include ECG data. The selected baselines do not include some reference models like SimCLR or MAEs.\n\n", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "# Response to reviewer ve81\n\n\n###### We thank the reviewer for the constructive feedback. We have uploaded a revision and used blue to mark the new changes. Our detailed responses are as follows.\n-----\n**Q1: The paper does not provide motivation for why the authors picked these special datasets. There is no discussion about large datasets like MIMIC-IV that include ECG data.**\n\nThanks. The datasets used in the paper are all large and widely used EEG/ECG datasets. Previous works (such as [1][2][3][4][5]) also use these datasets. Some (such as SHHS, CHB-MIT, Cardiology, PTB-XL) are from the open PhysioNet repository, some (such as TUEV, TUAB) are from Temple University open EEG corpus, and some (such as PREST, IIIC-Seizure) are proprietary datasets. In terms of the dataset size, SHHS (portion 1) has ~185 GB, PREST has ~320 GB, Cardiology has ~77 GB, TUAB has ~159 GB, TUEV has ~37 GB, etc. Also, the recommended MIMIC-IV-ECG dataset is officially under embargo (please check the MIMIC-IV-ECG website), and we are unable to provide additional results.\n\n[1] McCallan, Niamh, Scot Davidson, Kok Yew Ng, Pardis Biglarbeigi, Dewar Finlay, Boon Leong Lan, and James McLaughlin. \"Epileptic multi-seizure type classification using electroencephalogram signals from the Temple University Hospital Seizure Corpus: A review.\" Expert Systems with Applications (2023): 121040.\n\n[2] Yang, Chaoqi, M. Brandon Westover, and Jimeng Sun. \"Manydg: Many-domain generalization for healthcare applications.\" ICLR (2023).\n\n[3] Prasanna, J., M. S. P. Subathra, Mazin Abed Mohammed, Robertas Damaševičius, Nanjappan Jothiraj Sairamya, and S. Thomas George. \"Automated epileptic seizure detection in pediatric subjects of CHB-MIT EEG database—a survey.\" Journal of Personalized Medicine 11, no. 10 (2021): 1028.\n\n[4] Sridhar, Niranjan, Ali Shoeb, Philip Stephens, Alaa Kharbouch, David Ben Shimol, Joshua Burkart, Atiyeh Ghoreyshi, and Lance Myers. \"Deep learning for automated sleep staging using instantaneous heart rate.\" NPJ digital medicine 3, no. 1 (2020): 106.\n\n[5] Biswal, Siddharth, Haoqi Sun, Balaji Goparaju, M. Brandon Westover, Jimeng Sun, and Matt T. Bianchi. \"Expert-level sleep scoring with deep neural networks.\" Journal of the American Medical Informatics Association 25, no. 12 (2018): 1643-1650.\n\n**Q2: The selected baselines do not include some reference models like SimCLR or MAEs. How would the proposed model compare to these reference models.**\n\nThanks, this is a misunderstanding. Our paper aims to provide a *new biosignal encoder architecture* that can handle different signal formats (with missing, variable lengths, and mismatched channels) at once. Our idea is orthogonal to the design of SimCLR or MAE, and instead, BIOT can be a backbone for them under the self-supervised setting. Note that our BIOT encoder combined with a final prediction layer can be a supervised model and combined with SimCLR/MAE/NCE can be an unsupervised model. \n\nPer the reviewer's request, we use BIOT as the backbone and try SimCLR and MAE (conservative and aggressive version with different data augmentation rates) in the self-supervised setting (to replace the current NCE loss in Equation 4). We use TUEV and TUAB as the datasets, show the new results in Appendix C.5, and copy them here. Detailed result analysis can be found in Appendix C.5. *Note that we want to kindly remind the reviewer that this paper proposes a new biosignal encoder architecture but not a new self-supervised learning framework*.\n\n| Model | (TUAB) Balanced Acc. | (TUAB) AUC-PR | (TUAB) AUROC | (TUEV) Balanced Acc. | (TUEV) Coken’s Kappa | (TUEV) Weighted F1 |\n|------|---|---------------------|---|---|----|---------------------|\n| BIOT + NCE (PREST)                     | 0.7907 ± 0.0050     | 0.8752 ± 0.0051     | 0.8730 ± 0.0021     | 0.5207 ± 0.0285     | 0.4932 ± 0.0301     | 0.7381 ± 0.0169     |\n| BIOT + NCE (PREST+SHHS)                | 0.8019 ± 0.0021     | 0.8749 ± 0.0054     | 0.8739 ± 0.0019     | 0.5149 ± 0.0292     | 0.4841 ± 0.0309     | 0.7322 ± 0.0196     |\n| BIOT + SimCLR (PREST)                  | 0.7894 ± 0.0072     | 0.8681 ± 0.0053     | 0.8715 ± 0.0074     | 0.5113 ± 0.0331     | 0.4862 ± 0.0209     | 0.7310 ± 0.0244     |\n| BIOT + SimCLR (PREST+SHHS)             | 0.7852 ± 0.0096     | 0.8655 ± 0.0014     | 0.8694 ± 0.0075     | 0.5147 ± 0.0311     | 0.4829 ± 0.0157     | 0.7280 ± 0.0348     |\n| BIOT + conservative MAE (PREST)        | 0.7393 ± 0.0087     | 0.8347 ± 0.0076     | 0.8259 ± 0.0081     | 0.4760 ± 0.0328     | 0.4393 ± 0.0420     | 0.6831 ± 0.0112     |\n| BIOT + conservative MAE (PREST+SHHS)   | 0.7410 ± 0.0098     | 0.8312 ± 0.0082     | 0.8262 ± 0.0061     | 0.4601 ± 0.0238     | 0.4349 ± 0.0182     | 0.6797 ± 0.0189     |\n| BIOT + aggressive MAE (PREST)          | 0.7679 ± 0.0045     | 0.8591 ± 0.0086     | 0.8412 ± 0.0068     | 0.4929 ± 0.0416     | 0.4728 ± 0.0364     | 0.7130 ± 0.0216     |\n| BIOT + aggressive MAE (PREST+SHHS)     | 0.7692 ± 0.0011     | 0.8525 ± 0.0079     | 0.8459 ± 0.0043     | 0.4970 ± 0.0409     | 0.4686 ± 0.0136     | 0.7086 ± 0.0195     |\n\n\n\n**Q3: There are no scaling experiments which seems to be the most important finding in GPT-style models. If we want to create the equivalent of a GPT for biosignals, we need to source and combine probably hundreds of similar datasets and think hard about the proportion of each modality (EEG, movement, ECG etc). However, the authors left this for future work.**\n\nThanks. We have improved the Abstract and Introduction to clarify our contributions. As we mentioned earlier, *this paper barely provides a powerful encoder architecture that is able to incorporate different datasets in training and is flexible in generalization*. This work does not claim to provide a biosignal GPT (which requires huge time efforts and computing resources), and we leave it as a future work.\n\n----\nThanks again for the valuable comments. Hope our rebuttal has addressed all your concerns."}, {"review_id": "qgHTMUQSfT", "reviewer": "Reviewer_vQ39", "summary": "The authors present a general Transformer-based pipeline for learning on biosignals such as EEG, ECG and human activity sensor data. The proposed approach relies on a tokenization scheme that includes temporal and channel position information and a spectral representation of a segment of a single-channel time series. This allows learning and inference on examples with missing channels and/or time points. The impact of contrastive pretraining is evaluated on multiple modality-specific downstream tasks (EEG seizure detection, ECG arrythmia prediction, etc.), as well as the impact of missing channels/segments and of supervised pretraining. Results compare favorably to existing baselines.", "questions": "1. The use of an attention implementation with linear complexity is compelling, especially if the tokenization scheme yields long sequences. Can you give a sense of this length for the experiments of Table 2 and 3? Given the results are already above the baselines choosing linear attention might be a good tradeoff, but it would be interesting to present a comparison with a vanilla Transformer block for at least one of the experiments (both in terms of performance and of running time like in Table 7).\n2. What is the impact of using a spectral (FFT) representation rather than the time series segment itself as the input to the segment embedding FCN? The phase information that is discarded might actually be useful for some downstream tasks, e.g. when fine microstructure is relevant to a task or there are events/stimuli in a stimulation protocol.\n3. At line 212, it is said that \"the first three datasets are used entirely for unsupervised pre-training\"; there are no splitting information available for these datasets in the Appendix (and the code only defines a training dataloader) and so I understand that the hyperparameters were selected based on downstream task validation performance only. It would be interesting to see how pretext task performance relates to downstream task performance, and whether it is similar across modalities (however this is purely curiosity from my part, I don't think such an analysis is necessary for this submission).\n4. At line 184, T=0.2, but in the appendix (line 565) it is said to be T=2.\n5. The use of the word \"montages\" in an EEG context in Table 1 and in Appendix B is confusing. Typically \"montages\" refer to a fixed set of channel positions or derivations. Therefore, line 483 should instead read something like \"First, the 16 derivations ...\".", "limitations": "Yes.", "rating": 7, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "Originality: The paper proposes an original approach to feed multivariate time series data into a Transformer through a bespoke tokenization scheme. The combination with a contrastive unsupervised pretraining task and the applicability to multiple different modalities is also novel.\n\nQuality: The paper is technically sound, with results presented on multiple downstream tasks and different modalities. There are multiple analyses supporting the choice of hyperparameters.\n\nClarity: The paper is overall clear and well organized. The provided code is very well organized and clearly formatted.\n\nSignificance: the developed methodology is likely to be reused and could yield pretrained \"backbone\" models that can be finetuned on different tasks. Reported improvements on different baseline tasks also suggest this approach can outperform SOTA models without pretraining. Finally, the model's ability to naturally work despite missing channels and segments is compelling.", "weaknesses": "Quality: some of the core choices behind the methodology could be better supported/justified. For instance: the use of a linear complexity attention module (see question 1), the choice of a spectral representation for the segment embedding (Q2) and the use of a contrastive loss with channel/segment dropout augmentation (instead of a different unsupervised task, e.g. masked autoencoding) for unsupervised pretraining.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "# Response to reviewer vQ39\n-------\n##### We thank the reviewer for your appreciation and constructive comments. We have uploaded a revision and used blue to mark the new changes. \n**Q1: Can you give a sense of this length for the experiments of Table 2 and 3?**\n\nSure, it can be calculated from Table 1. For example, for the SHHS dataset, we use 0.5s as token overlap and 1s as the token length. Then, a data sample with 30s duration will have (30 * 2 - 1) * 2 channels = 118 tokens. For PREST, we use 0.5s as token overlap and 1s as token length. Then, a data sample with 10s duration will have (10 * 2 - 1) * 16 = 304 tokens. Sample durations are all given from the data resources.\n\n**Q2: It would be interesting to present a comparison with a vanilla Transformer block for at least one of the experiments (both in terms of performance and of running time like in Table 7).**\n\nSure, we provided the comparison in Appendix C.6 (and copyed the results below). Generally, we find that the BIOT model with vanilla Transformer costs a bit more time while the performance of using linear complexity Transformer and vanilla Transformer are very similar. The reason might be that biosignals, such as EEG, are known to have *low-rank structure naturally*, and thus the low-rank approximation of the quadratic attention (in linear complexity Transformer) will not lead to a noticeable performance drop.\n\n- On the CHM-MIT dataset\n| Method   | Balanced Acc.    | AUC-PR  | AUROC | Time / epoch (s) |\n|--|--|--|---|--|\n| BIOT (with linear Transformer)    | 0.6640   | 0.2573     | 0.8646   | 55.5780 |\n| BIOT (with naive Transformer)    | 0.6669     | 0.2493      | 0.8637   | 62.2877   |\n\n- On the IIIC Seizure dataset\n| Method    | Balanced Acc.  | Cohen’s Kappa | Weighted F1  | Time / epoch (s)   |\n|--|--|--|---|--|\n| BIOT (with linear Transformer)     | 0.5762  | 0.4932    | 0.5773       | 25.4500    |\n| BIOT (with naive Transformer)      | 0.5810     | 0.4994   | 0.5822      | 28.2942  |\n\n- On the TUAB dataset\n| Method    | Balanced Acc.     | AUC-PR    | AUROC    | Time / epoch  (s)  |\n|--|--|--|---|--|\n| BIOT (with linear Transformer)   | 0.7925  | 0.8707  | 0.8691      | 25.2788       |\n| BIOT (with naive Transformer)      | 0.7902   | 0.8673  | 0.8652      | 27.3423       |\n\n- On the TUEV dataset\n| Method | Balanced Acc. | Cohen’s Kappa  | Weighted F1  | Time / epoch (s)  |\n|--|--|--|---|--|\n| BIOT (with linear Transformer)    | 0.4682 | 0.4482  | 0.7085  | 8.1812   |\n| BIOT (with naive Transformer)     | 0.4693  | 0.4512  | 0.7104  | 8.5493  |\n\n**Q3: What is the impact of using a spectral (FFT) representation rather than the time series segment itself as the input to the segment embedding FCN? The phase information that is discarded might actually be useful for some downstream tasks.**\n\nThanks, this is a valuable question. In our experiments, we use the FFT representation since relevant tasks in the paper would benefit more from the spectral domain than the raw signal microstructures. However, if the phase information is important in other applications, our BIOT can be flexible to use raw time series as inputs.\n\n**Q4: It would be interesting to see how pretext task performance relates to downstream task performance, and whether it is similar across modalities (however this is purely curiosity from my part, I don't think such an analysis is necessary for this submission).**\n\nThanks for your question, and this would lead to interesting discussions. We need to consider two aspects when analyzing the generalization performance. **First**, whether the pre-trained data is semantically (in terms of the downstream task) close to the downstream task data. **Second**, whether the pretraining task itself is optimized well (this is the reviewer's question). Intuitively, only when the pretraining data is close to the downstream data and the pretraining task is optimized well, then the downstream performance can be greatly improved. \n\nActually, Figure 4 and Figure 5 can verify the first aspect, given that the second aspect is satisfied (the supervised pretraining models are all optimized well). From Figure 4 and Figure 5, we can see that pretraining on more similar datasets will generate better gain in downstream finetuning. For the second aspect, we conduct experiments in Appendix C.8 by loading the checkpoints of pretrained model from different epochs, which shows that the generalization performance will be better if the pretrained model is optimized better. The conclusion is consistent on both EEG and ECG datasets.\n\n\n- Load the checkpoints of pretrained BIOT on PREST at different epochs (1, 2, 3, 4, 5, 8, 10, 20) and finetuning on the IIIC Seizure dataset.\n| Checkpoint Epoch | 1 |2| 3| 4| 5| 8 | 10 | 20 |\n|----| ---- | ---- | ---- | ---- | ---- | ---- | ---- | ---- | \n| loss in pretraining | 17.32 | 6.19 | 1.94|  0.93 | 0.63 | 0.42 | 0.33 | 0.31 |\n| Weighted F1 when finetuning on IIIC Seizure | 0.5773 | 0.5815 |  0.5804 | 0.5822 | 0.5813 | 0.5835 | 0.5825 | 0.5828 | \n\n- Load the checkpoints of pretrained BIOT on Cardiology-12 at different epochs (1, 2, 3, 4, 5, 8, 10, 20) and finetuning on the PTB-XL dataset.\n| Checkpoint Epoch |  1 |2| 3| 4| 5| 8 | 10 | 20 |\n|--| -- | ---- | ---- | ---- | ---- | ---- | ---- | ---- |\n| loss in pretraining | 27.23 |  9.49 | 2.59 | 1.57 | 1.13 | 1.23| 1.24 | 1.01 |\n| Weighted F1 when finetuning on IIIC Seizure | 0.7493 | 0.7525 | 0.7631 | 0.7689 | 0.7645 | 0.7684 | 0.7652 | 0.7671 |\n\n\n**Q5: At line 184, T=0.2, but in the appendix (line 565) it is said to be T=2.**\n\nThanks for finding this typo. T=0.2 is correct.\n\n**Q6: The use of the word \"montages\" in an EEG context in Table 1 and in Appendix B is confusing. Typically \"montages\" refer to a fixed set of channel positions or derivations. Therefore, line 483 should instead read something like \"First, the 16 derivations ...\".**\n\nThanks for your suggestion. We have updated these places (Line 482, Line 500, Line 506, Line 512) in the paper.\n"}, {"review_id": "6kHd0QKusq", "reviewer": "Reviewer_XY8p", "summary": "Biological signals are crucial for clinical applications, but current models are specialized for specific settings such as sampling rate and duration. The authors propose a pre-trained model that enables cross-data training that addresses differences in sensor settings such as mismatched channels, variable sample lengths and sampling frequencies, and missing values. Compared to other baselines, their pretrained model improved balanced accuracy up to 7%.", "questions": "•\tWhat are the limitations of using language modeling algorithms for biosignal tasks and considering biosignals as \"sentences\"?\n•\tWhy would this tokenization scheme not work across domains (e.g. using Pre Trained EEG on ECG) as part of the unified biosignal model?\n•\tWhat are the hyperparameter tuning range?", "limitations": "•\tThe authors mention that performance for other types of tasks were not compared, but they also don’t discuss whether the tokenization scheme would affect the performance of those tasks differently.", "rating": 5, "confidence": 4, "soundness": 2, "presentation": 1, "contribution": 2, "strengths": "•\tThe paper considers many different situations of biosignal applications: supervised learning with regularly-formatted data, supervised learning with irregularly data, unsupervised learning, pre-training on other datasets.\n•\tThe authors apply many datasets in experiments, making the results convincing.\n", "weaknesses": "•\tThe proposed method lacks technical innovation. There isn’t significant technical contribution on top of Transformer or Vision-transformer (ViT). The majority of the paper is applying transformer to different application situations, but lacks the special model design for biosignals and its heterogeneity. While well suited for an application/health informatics venue the novelty does not extend sufficiently to NeurIPS levels.\n•\tThe paper does not have a strong motivation or clearly declare it. From the abstract, the work (pre-trainable foundational biosignal models) is motivated by the success of LLM, however, chasing a hot topic should not be the motivation of a solid research work. Instead, it’s more important to discuss how LLM and transformer can be related to biosignal data.\n•\tThe authors mention that mismatched channels, variable sample lengths, and prevalent missing values are unique challenges associated with biosignals, however, do not further explain why and how are they challenges.\n•\tEven though Transformer is widely applied in different application, recurrent model is still an important baseline model especially for time-series data.\n•\tI’m not sure if it’s appropriate to say “biosignals are more complicated” (Line 67). Other data modalities/applications may have their own challenges too.\n•\tThe paper is not well written, for example, many paragraphs in introduction do not have connections to the previous or later one.\n•\tMore than one tables are too big and out of the range.\n", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "# Response to reviewer XY8p\n----\n##### We thank the reviewer for the helpful feedback. We have uploaded a revision with the changes marked as blue. Our detailed responses are as follows:\n\n**Q1: What are the limitations of using language modeling algorithms for biosignal tasks and considering biosignals as \"sentences\"?**\n\nAs we mentioned in the paper, many biosignals could have mismatched channels, missing values, or variable lengths across different datasets, and thus directly applying language modeling algorithms or other sequence models on these settings are suboptimal or infeasible. *This is our motivation for developing the BIOT encoder architecture.*\n\n**Q2: There isn’t significant technical contribution on top of Transformer or Vision-transformer (ViT). The majority of the paper is applying transformer to different application situations, but lacks the special model design for biosignals and its heterogeneity.**\n\nThanks, this is a misunderstanding. Our contribution is that common Transformer models or ViT models cannot handle irregular biosignals (with mismatched channels, missing values, or variable lengths), while in this paper, we transform different signals into unified formats and thus enable the joint learning and generalizing to downstream applications across different datasets. Our model is designed specifically for biosignals: the tokenization transformation is designed based on multi-channel structures of biosignals, and the segmentation step also follows the frequency representations of biosignals.\n\n**Q3: The paper does not have a strong motivation or clearly declare it. From the abstract, the work (pre-trainable foundational biosignal models) is motivated by the success of LLM, however, chasing a hot topic should not be the motivation of a solid research work. Instead, it’s more important to discuss how LLM and transformer can be related to biosignal data.**\n\nThanks, to reduce the confusion, we have rephrased our Abstract during the rebuttal. In the original Abstract and Introduction, we have mentioned that this paper is motivated by the fact that many previous biosignal learning works cannot apply/generalize to other settings. This paper proposes a powerful encoder architecture that is able to incorporate different datasets in training and is flexible in generalization since it can handle mismatched channels, variable lengths, and missing values. The connection to previous text modeling (LLM) work is that Transformers can handle variable lengths, and thus we are motivated to design a tokenization trick to transform biosignals into variable-length \"sentences\".\n\n**Q4: The authors mention that mismatched channels, variable sample lengths, and prevalent missing values are unique challenges associated with biosignals, however, do not further explain why and how are they challenges. I’m not sure if it’s appropriate to say “biosignals are more complicated” (Line 67). Other data modalities/applications may have their own challenges too.**\n\nThanks. The original Introduction explained why and how these three are challenging (Line 44 - 53). In additional, there are already previous works discussing how to transform images (ViT), audios (Wave2Vec), and natural languages into a sentence, and this work proposes a way to transform multichannel biosignals into a sentence. We mention it is more challenging in the sense that previously there is no relevant study on biosignal transformation. Based on the reviewer's comments, we further rephrased this part (Line 67-68) during the rebuttal. \n\n**Q5: Even though Transformer is widely applied in different application, recurrent model is still an important baseline model especially for time-series data.**\n\nYes, two baselines included in this paper CNN-Transformer (Peh et al., 2022) and FFCL (Li et al., 2022) are based on LSTM or contain a LSTM component, and they are inferior to our proposed model under various evaluations.\n\n**Q6: The paper is not well written, for example, many paragraphs in introduction do not have connections to the previous or later one. More than one tables are too big and out of the range.**\n\nThanks, we have rephrased the relevant parts and adjusted the size of the tables.\n\n**Q7: Why would this tokenization scheme not work across domains (e.g. using Pre Trained EEG on ECG) as part of the unified biosignal model?**\n\nThanks. We did not try the tokenization scheme and do not know whether it works or not across domains. The current paper did not train EEG and ECG or other signal types together since different types are semantically different, such as in sampling frequencies, signal amplitudes, and time series microstructures. Also the corpus used for pre-training might also need to align with or remotely connected to the downstream tasks (for example, currently it is unclear how ECG signals can help EEG learning tasks). But, unifying different signals can be a promising direction as future work.\n\n**Q8: What are the hyperparameter tuning range?**\n\nThanks. The initial Appendix C.3 has reported our special hyperparameters ablation studies, including frequency choices, overlapping lengths, and token lengths. This ablation study can provide insights on how to select our newly introduced hyperparameters. Other hyperparameters, such as learning rate, batch size, L2 regularizers, are selected following common procedures based on the validation set.\n\n**Q9: The authors mention that performance for other types of tasks were not compared, but they also don’t discuss whether the tokenization scheme would affect the performance of those tasks differently.**\n\nSorry, we kindly ask the reviewer to clarify the question. We would be happy to address your concerns if more details are provided (such as Line number).\n\n----\nThanks again for your comments. Hope our rebuttal has addressed all your concerns."}, {"review_id": "EmZ4mDWnfD", "reviewer": "Reviewer_ykvi", "summary": "This paper proposes a model called Biosignal Transformer (BIOT) that allows cross-dataset learning with different number channels, sequence lengths, and missing values. BIOT consists of a tokenization module that transforms multi-channel signals into a sequence (“sentence”), and a Linear Transformer Encoder to learn latent representations from the tokenized sequence. BIOT can be used for a wide variety of tasks, including self-supervised pre-training, supervised training (without vs with missing values), and supervised pre-training. Experiments suggest that BIOT outperforms existing methods on various biosignal classification tasks.", "questions": "1. As mentioned above, I’m concerned about the scalability of BIOT to biosignals with long sequence lengths and large number of channels (which is the case for many real-world data). The datasets chosen have relatively short sequences and small number of channels. It would be great to see some experiments on the scalability. And its potential limitation should be discussed.\n2. Many types of biosignals have graph structures (e.g., brain signals like EEGs). Flattening the channels into a sequence may be suboptimal for capturing such data structures. This limitation should be discussed.\n3. Please provide more details about token embeddings. For example, how is the energy vector computed? What’s the dimensionality of the embedding table for channel embedding? What are the sinusoidal and cosine functions for positional embedding? If they follow previous studies, please provide citations.\n4. More ablation studies are needed to show the impact of each component in BIOT. In particular, ablations for 1) normalization in the biosignal tokenization module and 2) each of the token embedding. ", "limitations": "Some limitations are discussed in the paper. As mentioned above, please include additional discussion on the following limitations: 1) scalability to long sequences and large number of channels and 2) suboptimal for graph-structured data.", "rating": 5, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. While Transformers have been widely used for modeling biosignals, there is some originality in the design of the token embeddings (segment embedding, channel embedding, and positional embedding).\n2. Overall the methods are technically sound and easy to understand.\n3. BIOT tackles the following challenges in modeling biosignals across datasets: mismatched channels, variable lengths, and missing values. ", "weaknesses": "1. My major concerns about the method are: 1) its scalability to biosignals with long sequences and large number of channels and 2) spatial dependencies among channels are not appropriately modeled (e.g., graph structure of brain regions is not captured). These limitations should be investigated and discussed. \n2. More details about token embeddings are needed.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "# Response to reviewer ykvi\n------\n###### We thank the reviewer for the constructive feedback. We have uploaded a revision and used blue to mark the new changes. Our detailed responses are as follows.\n\n**Q1: My concern is on the scalability to biosignals with long sequences and large number of channels. It would be great to see some experiments on the scalability.**\n\nThanks for bringing this point. The model can already handle long recordings and multiple channels in the following ways: (i) BIOT uses a linear complexity Transformer, so that the complexity scales linearly with sample lengths and channel sizes; (ii) Users can also remove the token overlaps and enlarge the token sizes to reduce the token numbers (i.e., \"sentence\" length). \n\nThe BIOT model can better handle long recordings and multiple channels with minor adjustments. For long recordings, we could segment the recordings into 10-30s sessions, apply our BIOT on each session, and finally use a top-level LSTM or Transformer to learn sequence embedding. For multiple channels (more than 256 channels), we can group neighboring channels or symmetric channels and tokenize them together, which will greatly shrink the final \"sentence\" length. During the rebuttal, we added the scalability discussion in Appendix C.7.\n\nTo provide more insights on the scalability of our model, we have provided a comparsion between linear transformer and original transformer (in terms of performance and runnning time) with different sample size inputs in Appendix C.6.\n\n- On the CHM-MIT dataset\n| Method                                 | Balanced Acc.        | AUC-PR              | AUROC              | Time per epoch       |\n|---|----------------------|---------------------|--------------------|----------------------|\n| BIOT (with linear Transformer)      | 0.6640 ± 0.0037      | 0.2573 ± 0.0088     | 0.8646 ± 0.0030    | 55.5780 ± 0.4229     |\n| BIOT (with naive Transformer)       | 0.6669 ± 0.0112      | 0.2493 ± 0.0088     | 0.8637 ± 0.0030    | 62.2877 ± 0.3629     |\n\n- On the IIIC Seizure dataset\n| Method                                | Balanced Acc.        | Cohen’s Kappa        | Weighted F1          | Time per epoch         |\n|---------------------------------------|----------------------|----------------------|----------------------|------------------------|\n| BIOT (with linear Transformer)     | 0.5762 ± 0.0034      | 0.4932 ± 0.0046      | 0.5773 ± 0.0031      | 25.4500 ± 4.0835       |\n| BIOT (with naive Transformer)      | 0.5810 ± 0.0029      | 0.4994 ± 0.0030      | 0.5822 ± 0.0044      | 28.2942 ± 4.0835       |\n\n- On the TUAB dataset\n| Method                                | Balanced Acc.        | AUC-PR               | AUROC               | Time per epoch        |\n|---------------------------------------|----------------------|----------------------|---------------------|-----------------------|\n| BIOT (with linear Transformer)     | 0.7925 ± 0.0035      | 0.8707 ± 0.0087      | 0.8691 ± 0.0033     | 25.2788 ± 0.1200      |\n| BIOT (with naive Transformer)      | 0.7902 ± 0.0033      | 0.8673 ± 0.0068      | 0.8652 ± 0.0052     | 27.3423 ± 0.2174      |\n\n- On the TUEV dataset\n| Method                               | Balanced Acc.        | Cohen’s Kappa        | Weighted F1          | Time per epoch       |\n|--------------------------------------|----------------------|----------------------|----------------------|----------------------|\n| BIOT (with linear Transformer)    | 0.4682 ± 0.0125      | 0.4482 ± 0.0285      | 0.7085 ± 0.0184      | 8.1812 ± 0.1228      |\n| BIOT (with naive Transformer)     | 0.4693 ± 0.0204      | 0.4512 ± 0.0317      | 0.7104 ± 0.0224      | 8.5493 ± 0.1814      |\n\n**Q2: Spatial dependencies among channels are not appropriately modeled (e.g., graph structure of brain regions is not captured). Flattening the channels into a sequence may be suboptimal for capturing such data structures.**\n\nThanks for your question. The current model does not capture the graph structures of different channels (which is not the paper focus). However, whether or not to capture the spatial graphical information seems orthogonal to our BIOT model design. Our model can easily adopt designs from Transformer-based models if they have captured the channel graph structures. BIOT model can also add channel graph structures in other ways. For example, in addition to the segment/channel/positional embedding, our model can add a spatial embedding to the token embedding by utilizing channel graph representation learning. We have added the discussion in Appendix C.7.\n\n**Q3: How is the energy vector computed?**\n\nThanks. We take FFT of each segment (e.g., 1s) and the results of FFT is the energy vector, where each entry is the energy for one frequency band.\n\n**Q4: What’s the dimensionality of the embedding table for channel embedding?**\n\nThanks. The embedding table for channel embedding is a matrix, the row size is the channel number, and the col size is the embedding dimension.\n\n**Q5: What are the sinusoidal and cosine functions for positional embedding? If they follow previous studies, please provide citations.**\n\nSure, thanks for reminding us. We have added it (Line 133) during the rebuttal. The same sinusoidal and cosine functions are used in the original transformer paper: Attention is All Your Needed.\n\n**Q6: More ablation studies are needed to show the impact of each component in BIOT. In particular, ablations for (1) normalization in the biosignal tokenization module and (2) each of the token embedding.**\n\nSure, we have provided that in Appendix C.9. Different token embeddings are all important and are additively useful since they capture different information from the model: signal, frequency, and channel/spatial information. More details can refer to Appendix C.9.\n\n---\nThanks again for your time and valuable questions. We hope our new results and explanations can clear your concerns. We are happy to explain any other component of the model."}]

Author rebuttal:
We thank all the reviewers for your time and constructive feedback. During the rebuttal, we have prepared a revision and used blue to mark the new changes. 

Meta-review: Thank you for your submission. Reviewers agreed that the experimental approach was well written (vQ39, ykvi), the introduction and motivation were weaker (XY8p), with some methodological detail missing (ykvi). The use of multiple datasets and evaluations was appreciated by all reviewers, although these datasets appeared arbitrarily chosen (ve81). The rebuttal is weak for this latter point; datasets being popular and large is not motivating. Given the wealth of biosignal data available, selection of datasets to evaluate specific aspects of the encoder structure is warranted (e.g. scaling to long sequences via Holter monitor data would address concerns of reviewer ykvi). Reviewers found the new encoder structure of interest (ve81, ykvi). There was limited justification of why mismatched channels, missing values, and variable lengths are the defining challenges of biosignals (XY8p), as opposed to other clear issues (non-stationarity, multiple scales, varying sampling frequencies, FFT vs. raw signal, etc). The missing data approximations are unrealistic (XY8p). There is an implication that this encoder could work for all biosignals (ECG, EEG, ABP, EMG, EOG, CVP, ICP, respirometry, actimetry, PPG, etc), the evaluation would be stronger with more biosignal modalities included as it heavily focuses on EEG currently. Scaling the model remains an unaddressed challenge (ykvi), as the rebuttal experiments focus on the subsequent components (linear transformer) rather than the embedding approach itself.

On balance the work seems a worthwhile contribution with room for improvement in many areas.
Decision: Accept (poster)