SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Chaoqi Yang $^{1}$ , M. Brandon Westover $^{2,3}$ , Jimeng Sun $^{1}$
<sup>1</sup>University of Illinois Urbana-Champaign, <sup>2</sup>Harvard Medical School
$^{3}$ Beth Israel Deaconess Medical Center
{chaoqiy2}@illinois.edu
Biological signals, such as electroencephalograms (EEG), play a crucial role in numerous clinical applications, exhibiting diverse data formats and quality profiles. Current deep learning models for biosignals (based on CNN, RNN, and Transformers) are typically specialized for specific datasets and clinical settings, limiting their broader applicability. This paper explores the development of a flexible biosignal encoder architecture that can enable pre-training on multiple datasets and fine-tuned on downstream biosignal tasks with different formats.
To overcome the unique challenges associated with biosignals of various formats, such as mismatched channels, variable sample lengths, and prevalent missing values, we propose Biosignal Transformer (BIOT). The proposed BIOT model can enable cross-data learning with mismatched channels, variable lengths, and missing values by tokenizing different biosignals into unified "sentences" structure. Specifically, we tokenize each channel separately into fixed-length segments containing local signal features and then re-arrange the segments to form a long "sentence". Channel embeddings and relative position embeddings are added to each segment (viewed as "token") to preserve spatio-temporal features.
The BIOT model is versatile and applicable to various biosignal learning settings across different datasets, including joint pre-training for larger models. Comprehensive evaluations on EEG, electrocardiogram (ECG), and human activity sensory signals demonstrate that BIOT outperforms robust baselines in common settings and facilitates learning across multiple datasets with different formats. Using CHB-MIT seizure detection task as an example, our vanilla BIOT model shows  $3\%$  improvement over baselines in balanced accuracy, and the pre-trained BIOT models (optimized from other data sources) can further bring up to  $4\%$  improvements. Our repository is public at https://github.com/ycq091044/BIOT.
Biosignals, such as EEG and ECG, are multi-channel time series recorded at high sampling rates (e.g.,  $256\mathrm{Hz}$ ) in various healthcare domains, including sleep medicine, neurological and cardiovascular disease detection, and activity monitoring. Deep learning (DL) models have demonstrated impressive success in automating biosignal analysis across diverse applications (Yang et al., 2021), encompassing sleep stage classification (Biswal et al., 2018; Yang et al., 2021; Phan and Mikkelsen, 2022), emotion analysis via EEG (Zhang et al., 2020; Suhaimi et al., 2020), action and motor imagery recognition (Venkatachalam et al., 2020), acute stress detection through electrodermal activity (Greco et al., 2021), EEG-based seizure epilepsy classification (Yang et al., 2023; Jing et al., 2023), and ECG-driven cardiac arrhythmia detection (Isin and Ozdalili, 2017; Parvaneh et al., 2019).
Various deep learning methods have been developed for biosignal learning. Some works use 1D convolutional neural networks (CNN) on raw signals (Jing et al., 2023; Nagabushanam et al., 2020;
Dar et al., 2020), while others preprocess the data with short-time Fourier transform (STFT) and employ 2D CNN models on the resulting spectrogram (Yang et al., 2022; Kim et al., 2020; Cui et al., 2020). Researchers also segment the signal and use a CNN segment encoder with a downstream sequence model (Zhang et al., 2019; Biswal et al., 2018; Jing et al., 2020; Almutairi et al., 2021), such as Transformer or recurrent neural networks (RNN), to capture temporal dynamics. Other approaches involve ensemble learning, feature fusion from multiple encoders (Li et al., 2022), and multi-level transformers to encode spatial and temporal features across and within different channels (Lawhern et al., 2018; Song et al., 2021; Liu et al., 2021).
These models (Jing et al., 2023; Yang et al., 2021; Biswal et al., 2018; Kostas et al., 2021; Du et al., 2022; Zhang et al., 2022) predominantly focus on biosignal samples with fixed formats for specific tasks, while real-world data may exhibit mismatched channels, variable lengths, and missing values. In this paper, our objective is to devise a flexible training strategy that can handle diverse biosignal datasets with varying channels, lengths, and levels of missingness. For example, is it possible to transfer knowledge from abnormal EEG detection (a binary classification task with samples having 64 channels, 5-second duration, recorded at  $256\mathrm{Hz}$ ) to improve another EEG task, such as seizure type classification (a multi-class task with 16 channels and a 10-second duration at  $200\mathrm{Hz}$ )? In reality, such data mismatches often arise from varying devices, system errors, and recording limitations. Additionally, it is also important to explore the potential of utilizing various unlabeled data sources.
To apply existing deep learning models to such settings of different biosignals, significant data processing is required to align the formats across multiple datasets. This may involve truncating or padding signals for consistent lengths (Zhang et al., 2022), and imputing missing channels or segments (Bahador et al., 2021). Such practices, however, may introduce unnecessary noise and shift data distributions, leading to poor generalization performance. Developing a flexible and unified model that accommodates biosignals with diverse formats can be advantageous.
In our paper, we develop the biosignal transformer (BIOT, summarized in Figure 1), which, to the best of our knowledge, is the first multi-channel time series learning model that can handle biosignals of various formats. Our motivation stems from the vision transformer (ViT) (Dosovitskiy et al., 2020) and the audio spectrogram transformer (AST) (Gong et al., 2021). The ViT splits the image into a "sentence" of patches for image representation while the AST splits the audio spectrogram into a "sentence" for 1D audio representation. We are inspired by that these "sentence" structures combined with Transformer (Vaswani et al., 2017) can handle variable-sized inputs.
However, it is non-trivial to transform diverse biosignals of various formats into unified "sentence" structures. This paper proposes BIOT to solve the challenge by a novel biosignal tokenization module that segments each channel separately into tokens and then flattens the tokens to form consistent biosignal "sentences" (illustrated in Figure 2). With the design, our BIOT can enable the knowledge transfer cross different data in the wild and allow joint (pre-)training on multiple biosignal data sources seamlessly. Our contributions are listed below.
- Biosignal transformer (BIOT). This paper proposes a generic biosignal learning model BIOT by tokenizing biosignals of various formats into unified "sentences."
- Knowledge transfer across different data. Our BIOT can enable joint (pre-)training and knowledge transfer across different biosignal datasets in the wild, which could inspire the research of large foundation models for biosignals.
- Strong empirical performance. We evaluate our BIOT on several unsupervised and supervised EEG, ECG, and human sensory datasets. Results show that BIOT outperforms baseline models and can utilize the models pre-trained on similar data of other formats to benefit the current task.
2 BIOT: Biosignal Transformer
As shown in Figure 1, our BIOT encoder cascades two modules: (i) the biosignal tokenization module that tokenizes an arbitrary biosignal (variable lengths, different channels, and missing values) into a "sentence" structure. This design can potentially enable previous language modeling techniques (Devlin et al., 2018; Liu et al., 2019; OpenAI, 2023) to empower the current biosignal models; (ii) a linear transformer module that captures complex token interactions within the "sentence" while maintaining linear complexity. After that, we also discuss the application of BIOT in different real-world settings. For all the notations used in the paper, we summarize them in Appendix A.
Biosignal Transformer (BIOT) Encoder
(1) Supervised Learning
(2) Supervised Learning (with missing)
(3) Unsupervised Pre-training
(4) Supervised Pre-training
Figure 1: Biosignal Transformer (BIOT). (Upper) Given a new data sample, we initially perform data preprocessing (resampling, normalization, tokenization, and flattening) to create a biosignal "sentence" using the biosignal tokenization module. We then learn the complex interactions within the "sentence" through the linear transformer module. (Lower) BIOT encoder is versatile, enabling supervised learning on complete data, data with missing values, and pre-training and fine-tuning across diverse data formats and tasks.
2.1 Module 1: Biosignal Tokenization
Motivation. The goal of this paper is to model heterogeneous biosignals (e.g., EEG samples with different channels for different tasks) with a unified model. For example, common EEG samples (Lopez et al., 2015), such as those for seizure detection, are recorded at  $256\mathrm{Hz}$  in the international 10-20 system<sup>1</sup> for 10-second long (Klem et al., 1999). With standard 16 montage editing, the sample is essentially a multi-channel time-series, represented as a matrix of size (16, 2560). However, format mismatch may prevent the applications on other similar data, such as different sampling rate (e.g.,  $200\mathrm{Hz}$  vs  $256\mathrm{Hz}$ ) (Jing et al., 2023), mismatched channels (i.e., different datasets have their own novel channels), variable recording duration (i.e., 30s per sample vs. 10s) (Zhang et al., 2022), missing segments (i.e., part of the recording is damaged due to device error). Thus, existing models may fail to utilize the mismatched data from different datasets.
Our BIOT solves the above challenges of mismatches by the following steps. We assume one biosignal sample as  $\mathbf{S} \in \mathbb{R}^{I \times J}$ . Here,  $I$  as the channels and  $J = \text{duration}(s) \times \text{rate}(Hz)$  as the length.
- Resampling. We first resample all data to the same rate, denoted by  $r \in \mathbb{R}^+$  (such as  $200\mathrm{Hz}$ ), by linear interpolation. The common rate  $r$  could be selected following clinical knowledge of a certain biosignal. For example, the highest frequency of interest in both EEG and ECG signals is commonly around  $100\mathrm{Hz}$ , and thus  $200\mathrm{Hz}$  or  $250\mathrm{Hz}$  can be suitable for typical EEG or ECG applications, according to Nyquist-Shannon sampling theorem (Nyquist, 1928; Shannon, 1949).
- Normalization: To alleviate the unit difference and amplitude mismatch across different channels and datasets, we use the 95-percentile of the absolute amplitude to normalize each channel. Formally, each channel  $\mathbf{S}[i]$  is normalized by  $\frac{\mathbf{S}[i]}{\text{percentile}([|\mathbf{S}[i,1]|, |\mathbf{S}[i,2]|, \ldots, |\mathbf{S}[i,J]|])\%, 95\%}$ .
<sup>1</sup>https://en.wikipedia.org/wiki/10-20_system_(EEG)
Sample 1:
Sample 2:
Figure 2: Biosignal Tokenization (no segment overlap in the examples). Sample 1 has four channels (Fp1, Fp2, O1, and O2) for 5 seconds. We tokenize each channel into segments/tokens and then parameterize these 20 segments with three embeddings. On the right, we use different colors to represent the channels (blue-Fp1, brown-Fp2, green-O1, and yellow-O2). Sample 2 has mismatched channels (no O2), variable lengths (Fp1 and Fp2 are shorter and flipped), and missing values (in O1). Using our method, we can still tokenize Sample 2 in a comparable "sentence". On both samples, the peach orange area indicates the spatial- or temporal-relevant tokens w.r.t. the highlighted token.
- Tokenization: For handling length variation, we tokenize the recording of each channel into segments/tokens of length  $t \in \mathbb{R}^{+}$ , and neighboring tokens can have overlaps of length  $p \in \mathbb{R}$  ( $p < t$ , e.g.,  $t = r$  and  $p = 0.5r$  corresponds to 1s and 0.5s). Thus, the  $k$ -th token ( $k = 1, 2,.., K$ ) in the  $i$ -th channel can be represented by slicing  $\mathbf{S}[i, (t - p)(k - 1) : (t - p)(k - 1) + t]$ . The number of tokens  $K$  per channel is limited by the inequality:  $(t - p)(K - 1) + t \leq J$ . Here, the overlap  $p$  is essential to maintain the temporal information for shorter signals. For example, if the length of a signal is 3 seconds (i.e.,  $J = 3r$ ), the set  $t = r, p = 0$  only generates 3 tokens per channel, while the set  $t = r, p = 0.5r$  gives 5 tokens per channel (see Appendix C.3). In cases of missing values, we drop the corresponding tokens directly (as shown in Sample 2). Note that, our tokenization applies to each channel, separately, which is different from previous works (Almutairi et al., 2021; Du et al., 2022) that split all channels together (which cannot work on Sample 2).
- Flattening: We finally flatten tokens from all channels into a consistent "sentence".
The above steps are non-parametric. To study the effect of sampling rate  $r$ , token length  $t$ , and overlap  $p$ , we provide ablation studies and insights in Appendix C.3. In the following, we design the token embedding for the biosignal "sentence", which combines information from three aspects.
- Segment embedding. We leverage fast Fourier transform (FFT) to extract an energy vector for each token where every dimension indicates the energy of a specific frequency. A fully connected network (FCN) is then applied on the energy vector to transform it into the segment embedding. Directly modeling the raw time-series to learn a segment embedding can be another option.
- Channel embedding (spatial). We learn an embedding table for all different channels and add the corresponding channel embedding to the token. Each color represents one channel in Figure 2.
- **Positional embedding (temporal).** In biosignals, the token order within the channel captures temporal information. We thus add relative positional embedding to the token embedding by using the sinusoidal/cosine functions (Vaswani et al., 2017), which does not need learnable parameters.
The above three embeddings are summed to create the token embedding, which can effectively capture the segment features as well as the spatio-temporal features. Finally, we denote the tokenized "sentence" as  $\mathbf{X} \in \mathbb{R}^{N \times l_1}$  where  $N$  is the number of tokens and  $l_1$  is the embedding dimension.
2.2 Module 2: Linear Transformer
Transformer with linear complexity for long biosignal "sentence". Next, we want to leverage the Transformer model (Vaswani et al., 2017) for modeling the "biosignal sentence". However, our transformation may lead to long "sentences" due to multiple channels. For example, the "sentence"
of a 64-channel EEG signal for 20 seconds (without overlaps) may have  $64 \times 20 = 1280$  tokens, and longer with the overlaps. Given that the original Transformer model is known to have quadratic complexity in both time and space, we adopt the linear attention mechanism (Wang et al., 2020; Katharopoulos et al., 2020) for biosignal learning applications.
Formally, let us assume  $\mathbf{W}^K, \mathbf{W}^V, \mathbf{W}^Q \in \mathbb{R}^{l_1 \times l_2}$  be the key, value, and query matrices. Our self-attention module uses a rank- $d$  approximation for the softmax attention  $(N \times N)$  by reduced-rank parameter matrices  $\mathbf{E}^\top \in \mathbb{R}^{N \times d}, \mathbf{F} \in \mathbb{R}^{d \times N}$  (where  $d \ll N$ ). The output  $\mathbf{H} \in \mathbb{R}^{N \times l_2}$  is,
$$
\begin{array}{l} \mathbf {H} = \operatorname {A t t e n t i o n} \left(\mathbf {X} \mathbf {W} ^ {Q}, \mathbf {E} \mathbf {X} \mathbf {W} ^ {K}, \mathbf {F} \mathbf {X} \mathbf {W} ^ {V}\right) (1) \\ = \underbrace {\operatorname {s o f t m a x} \left(\frac {\left(\mathbf {X W} ^ {Q}\right) \left(\mathbf {E X W} ^ {K}\right) ^ {\top}}{\sqrt {l _ {2}}}\right)} _ {N \times d} \cdot \underbrace {\mathbf {F X W} ^ {V}} _ {d \times l _ {2}}. (2) \\ \end{array}
$$
Main components of this module include one linear self-attention layer and one fully connected network. To enable stable training, we add layer normalization (Ba et al., 2016) before each component, residual connection (He et al., 2016) after each component, and dropout (Srivastava et al., 2014) right after the self-attention (see Figure 1), which improves the convergence.
BIOT Encoder. An illustration of our proposed BIOT encoder is shown in Figure 1 (upper), which comprises the biosignal tokenization module and multiple blocks of linear transformer modules. We obtain the final biosignal "sentence" embedding by a mean pooling step over all tokens. Note that appending a classification [CLS] token at the beginning of the "sentence" (after Module 1) is also a common option. However, we find it yields a slightly worse performance in our application, and thus we use mean pooling in the experiments.
2.3 Biosignal Learning in the Wild
Our proposed BIOT encoder can be applied in various real-world biosignal applications (illustrated in the lower part of Figure 1). These applications include (1) standard supervised learning, (2) learning with missing channels or segments, (3) & (4) pre-training on one or more datasets, and fine-tuning on other similar datasets with different input formats.
(1) Supervised Learning is the most common setting in the previous literature (Jing et al., 2023; Biswal et al., 2018). With the BIOT encoder model, we finally apply an exponential linear unit (ELU) activation (Clevert et al., 2015) and a linear layer for classification tasks.
(2) Supervised Learning (with missing). Many real biosignal data have mismatched channels, missing segments, and variable lengths, which prevents the applications of existing models (Jing et al., 2023; Song et al., 2021). Flexible as our model is, BIOT can be applied in this setting using the same model structure as in (1).
(3) Unsupervised Pre-training. We can jointly pre-train a general-purpose BIOT encoder on multiple large unlabeled datasets. In the experiments, we pre-train an unsupervised encoder using 5 million resting EEG samples (16 channels, 10s,  $200\mathrm{Hz}$ ) and 5 million sleep EEG samples (2 channels, 30s,  $125\mathrm{Hz}$ ), which is later utilized to improve various downstream tasks.
For the unsupervised pre-training, we take the following steps (a diagram is shown in Figure 1).
- Assume  $\mathbf{S}$  is the original biosignal. We first randomly dropout part of its channels and dropout part of the tokens from the remaining channels, resulting in a perturbed signal  $\tilde{\mathbf{S}}$ .
- We then obtain the embeddings of  $\mathbf{S}$  and  $\tilde{\mathbf{S}}$  by the same BIOT encoder. To form the objective, we want to use the perturbed signal to predict the embedding of the original signal. Thus, an additional predictor (i.e., two-layer neural network) is appended for the perturbed signal following (Grill et al., 2020). We use  $\mathbf{Z}$  and  $\tilde{\mathbf{Z}}$  to denote the real embedding of  $\mathbf{S}$  and predicted embedding from  $\tilde{\mathbf{S}}$ .
$$
\mathbf {Z} = \operatorname {B I O T} (\mathbf {S}), \tilde {\mathbf {Z}} = \operatorname {p r e d i c t o r} (\operatorname {B I O T} (\tilde {\mathbf {S}})). \tag {3}
$$
- Finally, contrastive loss (He et al., 2020; Chen et al., 2020) is used on  $\mathbf{Z}$  and  $\tilde{\mathbf{Z}}$  to form the objective.
$$
\mathcal {L} = \operatorname {C r o s s E n t r o p y L o s s} \left(\operatorname {s o f t m a x} \left(\langle \mathbf {Z}, \tilde {\mathbf {Z}} ^ {\top} \rangle / T\right), \mathbf {I}\right). \tag {4}
$$
Here,  $T$  represents the temperature ( $T = 0.2$  throughout the paper) and  $\mathbf{I}$  is an identity matrix. In the implementation, we also apply sample-wise L2-normalization on  $\mathbf{Z}$  and  $\tilde{\mathbf{Z}}$  before matrix product.
(4) Supervised Pre-training aims to pre-train a model by supervised learning on one task and then generalize and fine-tune the encoder on a new task. The goal here is to transfer knowledge among different datasets and gain improvements on the new task compared to training from scratch. Our BIOT model allows the new datasets to have mismatched channels and different lengths.
This section shows the strong performance of BIOT on several EEG, ECG, and human sensory datasets. Section 3.2, 3.3 compare BIOT with baselines on supervised learning and learning with missing settings. Section 3.4, 3.5, 3.6 show the that BIOT can be flexibly pre-trained on various datasets (supervised or unsupervised) to improve a new task with different sample formats.
3.1 Experimental Setups
Biosignal Datasets. We consider the following datasets in the evaluation: (i) SHHS (Zhang et al., 2018; Quan et al., 1997) is a large sleep EEG corpus from patients aged 40 years and older. (ii) PREST is a large unlabeled proprietary resting EEG dataset; (iii) Cardiology (Alday et al., 2020) is a collection of five ECG datasets (initially contains six, and we exclude the PTB-XL introduced below). (iv) The CHB-MIT database (Shoeb, 2009) is collected from pediatric patients for epilepsy seizure detection. (v) IIIC Seizure dataset is from Ge et al. (2021); Jing et al. (2023) for detecting one of the six ictal-interictal-injury-continuum (IIIC) seizure patterns (OTH, ESZ, LPD, GPD, LRDA, GRDA); (vi) TUH Abnormal EEG Corpus (TUAB) (Lopez et al., 2015) is an EEG dataset that has been annotated as normal or abnormal; (vii) TUH EEG Events (TUEV) (Harati et al., 2015) is a corpus of EEG that contains annotations of segments as one of six sleep or resting event types: spike and sharp wave (SPSW), generalized periodic epileptiform discharges (GPED), periodic lateralized epileptiform discharges (PLED), eye movement (EYEM), artifact (ARTF) and background (BCKG); (viii) PTBLX (Wagner et al., 2020) is an ECG dataset with 12-lead recordings for diagnosis prediction, and we used it for arrhythmias phenotyping in this paper; (ix) HAR (Anguita et al., 2013) is a human action recognition dataset using smartphone accelerometer and gyroscope data.
Table 1: Dataset Statistics
Datasets	Type (subtype)	# Recordings	Rate	Channels	Duration	# Sample	Tasks
SHHS	EEG (sleep)	5,445	125Hz	C3-A2, C4-A1	30 seconds	5,093,522	Unsupervised pre-training
PREST	EEG (resting)	6,478	200Hz	16 montages	10 seconds	5,110,992	Unsupervised pre-training
Cardiology	ECG	21,264	500Hz	6 or 12 ECG leads	10 seconds	495,970	Unsupervised pre-training
CHB-MIT	EEG (resting)	686	256Hz	16 montages	10 seconds	326,993	Binary (seizure or not)
IIIC Seizure	EEG (resting)	2,702	200Hz	16 montages	10 seconds	165,309	Multi-class (6 seizure types)
TUAB	EEG (unknown)	2,339	256Hz	16 montages	10 seconds	409,455	Binary (abnormal or not)
TUEV	EEG (sleep and resting)	11,914	256Hz	16 montages	5 seconds	112,491	Multi-class (6 event types)
PTB-XL	ECG	21,911	500Hz	12 ECG leads	5 seconds	65,511	Binary (arrhythmias or not)
HAR	Wearable sensors	10,299	50Hz	9 coordinates	2.56 seconds	10,299	Multi-class (6 actions)