SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Dataset Processing. The first three datasets are used entirely for unsupervised pre-training. The next four datasets are used for supervised learning, and we used the common 16 bipolar montage channels in the international 10-20 system. For CHB-MIT (containing 23 patients), we first use patient 1 to 19 for training, 20,21 for validation, and 22,23 for test. Then, we flip the validation and test sets and conduct the experiments again, and we report the average performance on these two settings. For IIIC seizure, we divide patient groups into training/validation/test sets by  $60\%$ :  $20\%$ :  $20\%$ . For TUAB and TUEV, the training and test separation is provided by the dataset. We further divide the training patients into training and validation groups by  $80\%$ :  $20\%$ . For PTB-XL, we divide patient groups into training/validation/test sets by  $80\%$ :  $10\%$ :  $10\%$ . The train and test set of HAR is provided, and we further divide the test patients into validation/test by  $50\%$ :  $50\%$ . For all the datasets, after assigning the patients to either training, validation, or test groups, we will further split the patient's recording to samples, and the sample duration accords to the annotation files. The dataset statistics can be found in Table 1, and we provide more descriptions and processing details in Appendix B.1.
Baseline. We consider the following representative models: (i) SPaRCNet (Jing et al., 2023) is a 1D-CNN based model with dense residual connections, more advanced than the popular ConvNet (Schirrmeister et al., 2017), CSCM (Sakhavi et al., 2018); (ii) ContraWR's (Yang et al., 2021) encoder model first transforms the biosignals into multi-channel spectrogram and then uses 2D-CNN based ResNet (He et al., 2016); (iii) CNN-Transformer (Peh et al., 2022) is superior to CNN-LSTM models (Zhang et al., 2019); (iv) FFCL (Li et al., 2022) combines embeddings from CNN and LSTM
encoders for feature fusion; (v) ST-Transformer Song et al. (2021) proposes an multi-level EEG transformer for learning spatial (S) and temporal (T) features simultaneously, empirically better than EEGNet Lawhern et al. (2018). Our BIOT model trained from scratch is denoted by (vanilla).
**Environments and Settings.** The experiments are implemented by Python 3.9.12, Torch 1.13.1+cu117, Pytorch-lightning 1.6.4 on a Linux server with 512 GB memory, 128-core CPUs and eight RTX A6000 GPUs. All the models are optimized on training set and evaluated on the test set. The best model and hyperparameter combinations are selected based on the validation set. For Table 2 and Table 3, we obtain five sets of results with different random seeds and report the mean and standard deviation values. For Figure 3 and Figure 4, we report the results under three random seeds. More experimental and implementation details can refer to Appendix B.2.
3.2 Setting (1) - standard supervised learning
This section shows that BIOT is comparable or better than baselines in the supervised learning settings.
- Four EEG Tasks. Both CHB-MIT and TUAB are designed to predict binary output, and we use binary cross entropy (BCE) for TUAB and the focal loss (Lin et al., 2017) for CHB-MIT due to its imbalances (around  $0.6\%$  positive ratio in training set). We use balanced accuracy (Balanced Acc.), area under precision-recall curve (AUC-PR) and AUROC as the metrics. Both IIIC Seizure and TUEV are multi-class classification tasks with cross entropy loss. We employ Balanced Acc., Cohen's Kappa, and Weighted F1 as the multi-class evaluation. To save space, we only show the performance on CHB-MIT and IIIC Seizure in Table 2 and move the other two to Appendix C.1.
- ECG and Sensory Tasks. PTB-XL is formulated as a binary classification on detecting arrhythmias phenotypes. We use the BCE loss and binary evaluation metrics. HAR (classifying actions) uses the cross entropy loss and is evaluated by multi-class metrics. Results are reported in Table 3. We provide the running time comparison of all experiments in Appendix C.4.
Table 2 and 3 show that our model has superior performance over baselines in most tasks, especially on CHB-MIT, IIIC Seizure, and HAR. The reason might be that the frequency features are more useful in these three tasks as our BIOT extracts the main features from spectral perspective. SPaRCNet is a strong model among all the baselines except on the CHB-MIT task. The model might be vulnerable in the imbalanced classification setting even with the focal loss. The pre-training models at the end of the tables will be introduced and explained in Section 3.4, 3.6.
Table 2: EEG classification tasks (Results of TUAB and TUEV are in Appendix C.1)
Models	CHB-MIT (seizure detection)	IIIC Seizure (seizure type classification)
Balanced Acc.	AUC-PR	AUROC	Balanced Acc.	Cohen's Kappa	Weighted F1
SPaRCNet (Jing et al., 2023)	0.5876 ± 0.0191	0.1247 ± 0.0119	0.8143 ± 0.0148	0.5546 ± 0.0161	0.4679 ± 0.0228	0.5569 ± 0.0184
ContraWR (Yang et al., 2021)	0.6344 ± 0.0002	0.2264 ± 0.0174	0.8097 ± 0.0114	0.5519 ± 0.0058	0.4623 ± 0.0148	0.5486 ± 0.0137
CNN-Transformer (Peh et al., 2022)	0.6389 ± 0.0067	0.2479 ± 0.0227	0.8662 ± 0.0082	0.5476 ± 0.0103	0.4481 ± 0.0139	0.5346 ± 0.0127
FFCL (Li et al., 2022)	0.6262 ± 0.0104	0.2049 ± 0.0346	0.8271 ± 0.0051	0.5617 ± 0.0117	0.4704 ± 0.0130	0.5617 ± 0.0171
ST-Transformer (Song et al., 2021)	0.5915 ± 0.0195	0.1422 ± 0.0094	0.8237 ± 0.0491	0.5423 ± 0.0056	0.4492 ± 0.0056	0.5440 ± 0.0014
(Vanilla) BIOT	0.6640 ± 0.0037	0.2573 ± 0.0088	0.8646 ± 0.0030	0.5762 ± 0.0034	0.4932 ± 0.0046	0.5773 ± 0.0031
Pretrained BIOT (PREST)	0.6942 ± 0.0431	0.3072 ± 0.1187	0.8679 ± 0.0106	0.5787 ± 0.0066	0.4980 ± 0.0054	0.5828 ± 0.0049
Pretrained BIOT (PREST+SHHS)	0.6788 ± 0.0036	0.3090 ± 0.0003	0.8752 ± 0.0022	0.5800 ± 0.0004	0.5040 ± 0.0041	0.5878 ± 0.0015
Pretrained BIOT (6 EEG datasets)	0.7068 ± 0.0457	0.3277 ± 0.0460	0.8761 ± 0.0284	0.5779 ± 0.0087	0.4949 ± 0.0103	0.5737 ± 0.0088
1. All models use the same training set of the task, while the pre-trained BIOT models are initially pre-trained on other data sources (see Section 3.4, 3.6).
2. Bold for the best model (trained from scratch) and [box] for the best pre-trained models. Running time comparison is in Appendix C.4.
Table 3: ECG and human activity sensory classification tasks
Models	PTB-XL (arrhythmias phenotype prediction)	HAR (huamn action recognition)
Balanced Acc.	AUC-PR	AUROC	Balanced Acc.	Cohen's Kappa	Weighted F1
SPaRCNet (Jing et al., 2023)	0.8275 ± 0.0047	0.9040 ± 0.0067	0.7550 ± 0.0073	0.9371 ± 0.0160	0.9236 ± 0.0189	0.9365 ± 0.0155
ContraWR (Yang et al., 2021)	0.7532 ± 0.0561	0.7549 ± 0.0164	0.5258 ± 0.1190	0.9068 ± 0.0164	0.8879 ± 0.0201	0.9055 ± 0.0182
CNN-Transformer (Peh et al., 2022)	0.6650 ± 0.0459	0.7175 ± 0.0558	0.4996 ± 0.0936	0.8690 ± 0.0839	0.8273 ± 0.0953	0.8352 ± 0.1166
FFCL (Li et al., 2022)	0.7034 ± 0.0052	0.7088 ± 0.0053	0.5127 ± 0.0051	0.8519 ± 0.0148	0.8216 ± 0.0177	0.8508 ± 0.0138
ST-Transformer (Song et al., 2021)	0.7238 ± 0.0083	0.7775 ± 0.0153	0.6003 ± 0.0179	0.9336 ± 0.0063	0.9213 ± 0.0076	0.9337 ± 0.0068
(Vanilla) BIOT	0.8315 ± 0.0008	0.8978 ± 0.0020	0.7493 ± 0.0167	0.9461 ± 0.0134	0.9351 ± 0.0160	0.9458 ± 0.0136
Pretrained BIOT (Cardiology-6)	0.8350 ± 0.0073	0.9128 ± 0.0094	0.7671 ± 0.0116	/	/	/
Pretrained BIOT (Cardiology-12)	0.8421 ± 0.0030	0.9221 ± 0.0075	0.7659 ± 0.0076	/	/	/
* Bold for the best model. All models use the same training set of the task. The Pretrained B10T (Cardiology-6) and Pretrained B10T (Cardiology-12) are pre-trained on Cardiology data (see Section 3.4), and they do not apply to HAR data (due to different biosignal types).
Figure 3: Supervised learning with missing channels or segments (on TUEV and IIIC Seizure)
3.3 Setting (2) - learning with missing channels and segments
The section simulates the TUEV dataset (16 channels and 5s per sample) and IIIC Seizure (16 channels and 10s per sample) to mimic the setting of supervised learning with missing channels and segments, which shows the strong performance of BIOT. We consider three missing cases:
- Missing segments: Randomly mask out  $a$  segments (each segment spans for 0.5 seconds),  $a = 0,1,2,3,4,5$  with equal probability. The segment masking is applied separately for each channel.
- Missing channels: Randomly mask out  $b$  channels,  $b = 0,1,2,3,4$  with equal probability. We assume that the masking will not alter the underlying labels (the same assumption for other cases).
- Missing both channels and segments: Combine both masking strategies simultaneously.
To enable the baseline models compatible with the setting, we use all zeros to impute the masked regions. The comparison is plotted in Figure 3, which shows that (i) all models decrease the performance with more missings while BIOT and the pre-trained BIOT are less impacted (especially on Kappa and Weighted F1); (ii) "Missing channels" affects the performance more than "Missing segments", which makes sense as segment masking still preserves information from all channels.
3.4 Setting (3) - unsupervised pre-training
This section shows that BIOT enables unsupervised pre-training on existing datasets with various formats. Note that, all the pre-trained models in this paper have a similar scale ( $\sim 3.3$  million parameters). We defer the study of "scaling effect" of pre-trained BIOT to future work.
- Pre-trained (PREST): This model is pre-trained on 5 million resting EEG samples (PREST) with 2,048 as the batch size. We save the pre-trained model at the 100-th epoch.
- Pre-trained (PREST+SHHS): This model is jointly pre-trained on 5M PREST and 5M SHHS EEG samples. Though two datasets have different sample formats, our model is able to use them together. Also, we use 2048 as the batch size and save model at the 100-th epoch.
- Pre-trained (Cardiology-12) is jointly pre-trained on raw data of five datasets in Cardiology corpus (details in Appendix B.1). We use 1024 as batch size and save model at the 100-th epoch.
- Pre-trained (Cardiology-6) is pre-trained similarly as Pre-trained (Cardiology-12), while we only utilize the first 6 ECG leads. By contrast, Pre-trained (Cardiology-12) uses full 12 leads.
We fine-tune the first two pre-trained EEG models on four EEG tasks and append the results to Table 2 (also in Appendix C.1). We fine-tune the last two pre-trained ECG models on PTB-XL datasets in Table 3. The pre-trained models can be seamlessly applied to various downstream tasks with different sample formats. Results show that the pre-trained models can improve the performance consistently after fine-tuning on the training set of downstream tasks in Table 2 and Table 3.
3.5 Setting (4) - supervised pre-training on other tasks
This section shows that BIOT allows knowledge transfer from one task to another similar task with different sample formats. We pre-train on the training set of CHB-MIT, IIIC Seizure, TUAB and fine-tunes on TUEV (which has 16 channels and 5s duration). All datasets use  $200\mathrm{Hz}$  sampling rate. We design three sets of configurations for the pre-trained datasets: Format (i) uses the first 8 channels and 10s duration; Format (ii) uses the full 16 channels but only the first 5s recording; Format (iii) uses full 16 channels and full 10s recording. During fine-tuning, we then remove the prediction layers from these pre-trained model and add a new prediction layer to fit the TUEV dataset.
Figure 4: Fine-tuned on TUEV from different supervised pre-trained models (best number in **bold**). Similar supervised fine-tuning analysis on CHB-MIT dataset is shown in Appendix C.2.
The fine-tuning results on TUEV are shown in Figure 4 where we also add the vanilla BIOT for reference. We find that (i) the model pre-trained on IIIC Seizure and TUAB are generally beneficial for the event classification task on TUEV. The reason might be that TUAB and TUEV are both recorded from Temple University and share some common information, while IIIC seizure and TUEV are both related to seizure detection and may share some latent patterns. (ii) More pre-training data leads to better results in the downstream task: though the pre-training configuration (16 channels, 5 seconds) aligns better with the TUEV data formats, the results show that configuration of (16 channels, 10 seconds) encodes longer duration and works consistently better. (iii) Compared to the TUEV results in Appendix C.1, we also find that oftentimes the supervised pre-training (e.g., on IIIC seizure or TUAB) can be more effective than unsupervised pre-training (e.g., on SHHS and PREST).
3.6 Pre-trained on all EEG datasets
In this section, we show that BIOT can leverage all six EEG resources considered in the paper. We obtain a Pre-trained (6 EEG datasets) model by loading the Pre-trained (PREST+SHHS) model and further train it on the training sets of CHB-MIT, IIIC Seizure, TUAB, and TUEV. We add separate classification layers for four tasks. Essentially, this model is pre-trained on all six EEG datasets. To use the model, we still fine-tune it on the training set of downstream tasks and append the results to Table 2 and Appendix C.1. Apparently, Pre-trained (6 EEG datasets) outperforms the vanilla BIOT and is better than the unsupervised and the supervised pre-trained BIOT models in most cases.
4 Conclusions and Discussions
This paper proposes a new biosignal transformer model (BIOT) that learns embeddings for biosignals with various formats. BIOT can enable effective knowledge transfer across different data and allow joint training on multiple sources. We conduct extensive evaluations on nine biosignal datasets and show that our BIOT is flexible and effective in various cross-data learning settings. Future efforts can explore the different types of biosignals and pre-training an all-in-one unified biosignal model. We hope our work can inspire more follow-up researches of large foundational models for biosignals.
Yang, C., Xiao, D., Westover, M. B., and Sun, J. (2021). Self-supervised eeg representation learning for automatic sleep staging. arXiv preprint arXiv:2110.15278.
Zhang, G.-Q., Cui, L., Mueller, R., Tao, S., Kim, M., Rueschman, M., Mariani, S., Mobley, D., and Redline, S. (2018). The national sleep research resource: towards a sleep data commons. Journal of the American Medical Informatics Association, 25(10):1351-1358.
Zhang, R., Zong, Q., Dou, L., and Zhao, X. (2019). A novel hybrid deep learning scheme for four-class motor imagery classification. Journal of neural engineering, 16(6):066004.
Zhang, X., Zhao, Z., Tsiligkaridis, T., and Zitnik, M. (2022). Self-supervised contrastive pre-training for time series via time-frequency consistency. arXiv preprint arXiv:2206.08496.
Zhang, Y., Chen, J., Tan, J. H., Chen, Y., Chen, Y., Li, D., Yang, L., Su, J., Huang, X., and Che, W. (2020). An investigation of deep learning models for eeg-based emotion recognition. Frontiers in Neuroscience, 14:622759.
A Notaion Table
The listed the notations used in the main text in the following Table 4.
Table 4: Notations used in BIOT
Symbols	Descriptions
I∈N+	the number of channels in biosignal, such as 16
J∈R+	the length of biosignals, such as 10s × 200Hz (use a complete sample for ease of notation)
S∈RI×J	the multichannel biosignal
S[i]∈RJ	the i-th channel of the biosignal.
r∈R+	the sampling rate, such as 200Hz
t∈R+	the length of the biosignal token, such as 1s × 200Hz
p∈R+	the overlap length between two neighboring tokens, such as 0.5s × 200Hz, t > p
k∈N+	the k-th token in the i-th channel is denoted by S[i, (t - p)(k - 1): (t - p)(k - 1) + t] which has length t
K∈N+	max number of tokens in one channel
N∈N+	the number of tokens in tokenized biosignal "sentence"
l1∈N+	the dimension of the token embedding
X∈RN×l1	the tokenized biosignal "sentence"
l2∈N+	the new dimensions of tokens after self-attention module
W^K, W^V, W^Q \in R^{l_1 \times l_2}	the key, value, and query matrices in self-attention module
E, F \in Rd×N	the low-rank projection matrices
d∈N+	the reduced rank for self-attention matrices, d << N
H \in RN×l2	the output of the self-attention module
\(\tilde{\mathbf{S}} \in \mathbb{R}^{I \times J}\)	the perturbed biosignal sample
Z, Z	the real and predicted embeddings of S
T	the temperature hyperparameter in the contrastive loss
I	the identity matrix in the contrastive loss, has the size of (batch size, batch size)
a∈[0,1,2,3,4,5]	the discrete uniformly distributed variable for the number of masked segments in Section 3.3
b∈[0,1,2,3,4]	the discrete uniformly distributed variable for the number of masked channels in Section 3.3
B Details of Datasets and Experimental Settings
B.1 More for Datasets and Processings
We provide more descriptions on each dataset in this section.
For EEG datasets. First, the 16 derivations (in 10-20 international system) are "FP1-F7", "F7-T7", "T7-P7", "P7-O1", "FP2-F8", "F8-T8", "T8-P8", "P8-O2", "FP1-F3", "F3-C3", "C3-P3", "P3-O1", "FP2-F4", "F4-C4", "C4-P4", "P4-O2".
- Sleep Heart Health Study (SHHS) (Zhang et al., 2018; Quan et al., 1997) is a multi-center cohort study from the National Heart Lung & Blood Institute assembled to study sleep-disordered breathing, which contains 5,445 recordings. The data is accessible upon request in their website  ${}^{2}$  . Each recording has 14 Polysomnography (PSG) channels, and the recording frequency is  ${125.0}\mathrm{\;{Hz}}$  . We use the C3/A2 and C4/A1 EEG channels. The dataset is released with sleep annotations. We use the existing codes  ${}^{3}$  and split each recordings into 30-second samples. In this study, we use SHHS samples for unsupervised pre-training without the original labels.
- PREST is a private dataset recorded in hospital sleep lab, primarily for seizure and abnormal EEG detection purpose (such as spikes). The local IRB waived the requirement for informed consent for this retrospective analysis of EEG data. We follow the clinician's instructions and split each recordings into 10 seconds without labels. In the experiment, we use it for EEG model pre-training.
- The CHB-MIT database  ${}^{4}$  (Shoeb,2009) is publicly available,which is collected at the Children's Hospital Boston, consists of EEG recordings from pediatric subjects with intractable seizures. The dataset is under Open Data Commons Attribution License v1.0  ${}^{5}$  and is used to predict whether the EEG recordings contain seizure signals. Each recording initially contains 23 bipolar channels and
$^{2}$ https://sleepdata.org/datasets/shhs
<sup>3</sup>https://github.com/ycq091044/ContraWR/tree/main/preprocess
<sup>4</sup>https://physionet.org/content/chbmit/1.0.0/
<sup>5</sup>https://physionet.org/content/chbmit/view-license/1.0.0/
we select the 16 standard derivations in the experiments. We utilize the existing preprocessing  ${}^{6}$  and follow the typical practices to further split each recordings into 10-second non-overlapping samples by default. Since the dataset is highly imbalanced, we use 5 seconds as overlaps to split the seizure regions (which could potentially double the positive samples). After processing, the positive ratio in the training set is around  ${0.6}\%$  .
- IIIC Seizure is requested from Ge et al. (2021); Jing et al. (2023), and we follow the license and usage statements in Jing et al. (2023). The samples follow 16 derivations and span 10-second signals at  $200\mathrm{Hz}$ . This dataset is used for predicting one of the six classes: lateralized periodic discharges (LPD), generalized periodic discharges (GPD), lateralized rhythmic delta activity (LRDA), generalized rhythmic delta activity (GRDA), Seizure types, and Other.
- TUH Abnormal EEG Corpus (TUAB) (Lopez et al., 2015) and TUH EEG Events (TUEV) (Harati et al., 2015) is accessible upon request at Temple University Electroencephalography (EEG) Resources  $^{7}$ . We process both datasets to follow the 16 EEG derivations.
For ECG datasets. We use the Cardiology collection to pre-train the ECG models and apply it on downstream supervised PTB-XL task.
- The Cardiology collection (Alday et al., 2020) is publicly available at physionet  $^{8}$ , which was used in the PhysioNet/Computing in Cardiology Challenge 2020. This collection is under Creative Commons Attribution 4.0 International Public License  $^{9}$ . In this study, we use five sets from the training portion of the collection (It has in total six sets. Another one overlaps with the PTB-XL dataset, and thus we drop it for pre-training), which contains recordings from CPSC2018 (6,877 recordings), CPSC2018Extra (China 12-Lead ECG Challenge Database – unused CPSC 2018 data, 3,453 recordings), St Petersburg Incart (12-lead Arrhythmia Database, 74 recordings), PTB (Diagnostic ECG Database, 516 recordings), Georgia (12-Lead ECG Challenge Database, 10,344 recordings). For preprocessing, we extract 10-second samples from each recording with 0.5s as the overlapping window (for obtaining more unsupervised trianing corpus). All the samples are merged together as an unsupervised pre-training ECG corpus of nearly 0.5 million samples. We pre-train a Pre-trained BIOT (Cardiology-12) on all the channels and a Pre-trained BIOT (Cardiology-6) on the first 6-channels of all samples. The sample sizes are different from the below PTB-XL dataset.
- Physikalisch-Technische Bundesanstalt (PTB-XL)  $^{10}$  (Wagner et al., 2020) is a publicly available large dataset of 12-lead ECGs from 18885 patients. It is under the Creative Commons Attribution 4.0 International Public License  $^{11}$ . The raw waveform data was annotated by up to two cardiologists, who assigned potentially multiple ECG statements to each record up to 27 diagnoses: 1:1st degree AV block, 2:Atrial fibrillation, 3:Atrial flutter, 4:Bradycardia, 5:Complete right bundle branch block, 6:Incomplete right bundle branch block, 7:Left anterior fascicular block, 8:Left axis deviation, 9:Left bundle branch block, 10:Low QRS voltages, 11:Nonspecific intraventricular conduction disorder, 12:Pacing rhythm, 13:Premature atrial contraction, 14:Premature ventricular contractions, 15:Prolonged PR interval, 16:Prolonged QT interval, 17:Q wave abnormal, 18:Right axis deviation, 19:Right bundle branch block, 20:Sinus arrhythmia, 21:Sinus bradycardia, 22:Sinus rhythm, 23:Sinus tachycardia, 24:Supraventricular premature beats, 25:T wave abnormal, 26:T wave inversion, 27:Ventricular premature beats. We following clinical knowledges and further groups them into six broader categories: Arrhythmias, Bundle branch blocks and fascicular blocks, Axis deviations, Conduction delays, Wave abnormalities, Miscellaneous. Each recordings can be associated to multiple categories. In this paper, we conduct the "Arrhythmias" phenotyping prediction task. If the recordings have at least one diagnosis belonging to the Arrhythmias group, then we label them as positive, otherwise as negative.
For human activity sensory data. Human activity recognition (HAR) dataset  $^{12}$  (Anguita et al., 2013) is publicly available at UCI machine learning repository. The data is collected from smartphone accelerometer and gyroscope data with 3D coordinates to detect six actions: walking, walking
$^{6}$ https://github.com/bernia/chb-mit-scalp
<sup>7</sup>https://isip.piconepress.com/projects/tuh_eeg/html/downloads.shtml
<sup>8</sup>https://physionet.org/content/challenge-2020/1.0.2/
<sup>9</sup>https://physionet.org/content/challenge-2020/view-license/1.0.2/
$^{10}$ https://physionet.org/content/ptb-xl/1.0.1/
11 https://physionet.org/content/ptb-xl/view-license/1.0.1/
<sup>12</sup>https://archive.ics.uci.edu/ml/datasets/human+activity+recognition+using+smartphones
upstairs, walking downstairs, sitting, standing, laying. The samples are already splitted and provided in the original datasets.
B.2 More for Experimental Settings
For model implementation, the SPaRCNet code is requested from the authors (Jing et al., 2023), the ContraWR code is downloaded and modified upon the github  $^{13}$ , CNN-Transformer is easily implemented following the Fig. 3 of the original paper (Peh et al., 2022), FFCL (Li et al., 2022) combines a CNN model and a LSTM model for learning separate representations and then merges them before the final prediction layer, the implementation of ST-Transformer refer to this repo  $^{14}$ . The linear-complexity attention module is referred to this repo  $^{15}$  in our BIOT implementation.