SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Table 2: In-domain setting of forecasting the future  $O$  time points based on the past 336 time points. All results are averaged from 4 different choices of  $O \in \{96,192,336,720\}$ . A smaller MSE or MAE indicates a better prediction. See Appendix E for full results.
Models	SimMTM	Random init.	Ti-MAE [21]	TST [56]	LaST [42]	TF-C [57]	CoST [46]	TS2Vec [55]
Metric	MSE	MAE	MSE	MAE	MSE	MAE	MSE	MAE	MSE	MAE	MSE	MAE
ETTh1	0.409	0.428	0.431	0.448	0.423	0.446	0.466	0.462	0.474	0.461	0.527	0.513
ETTh2	0.353	0.390	0.395	0.427	0.380	0.386	0.404	0.421	0.449	0.459	0.692	0.724
ETTm1	0.348	0.385	0.356	0.387	0.366	0.391	0.373	0.389	0.398	0.398	0.496	0.474
ETTm2	0.263	0.320	0.279	0.336	0.267	0.325	0.297	0.347	0.265	0.327	0.465	0.562
Weather	0.230	0.271	0.239	0.275	0.234	0.265	0.239	0.276	0.232	0.261	0.286	0.349
Electricity	0.162	0.256	0.212	0.300	0.205	0.296	0.209	0.289	0.186	0.274	0.363	0.432
Traffic	0.392	0.264	0.490	0.316	0.475	0.310	0.586	0.362	0.713	0.397	0.717	0.456
Avg	0.308	0.331	0.343	0.356	0.336	0.346	0.368	0.364	0.388	0.368	0.507	0.501
Table 3: Cross-domain setting of forecasting the future  $O$  time points based on the past 336 time points. All results are averaged from 4 different choices of  $O \in \{96,192,336,720\}$ . A lower MSE or MAE indicates a better prediction. Full results are in Appendix E.
Models	SimMTM	Ti-MAE [21]	TST [56]	LaST [42]	TF-C [57]	CoST [46]	TS2Vec [55]
Metric	MSE	MAE	MSE	MAE	MSE	MAE	MSE	MAE	MSE	MAE	MSE	MAE	MSE
ETTh2 → ETTh1	0.415	0.430	0.466	0.456	0.469	0.459	0.443	0.471	0.635	0.634	0.428	0.433	0.517	0.486
ETTm1 → ETTh1	0.422	0.430	0.495	0.469	0.475	0.463	0.426	0.441	0.700	0.702	0.620	0.541	0.484	0.482
ETTm2 → ETTh1	0.428	0.441	0.464	0.456	0.453	0.450	0.503	0.507	1.091	0.814	0.598	0.548	0.616	0.550
Weather → ETTh1	0.456	0.467	0.462	0.464	0.465	0.456	-	-	-	-	0.518	0.487	0.463	0.460
ETTh1 → ETTm1	0.346	0.384	0.360	0.390	0.373	0.393	0.353	0.390	0.746	0.652	0.370	0.393	0.699	0.557
ETTh2 → ETTm1	0.365	0.384	0.383	0.402	0.391	0.409	0.475	0.489	0.750	0.654	0.363	0.387	0.694	0.557
ETTm2 → ETTm1	0.351	0.383	0.390	0.410	0.382	0.402	0.414	0.464	0.758	0.699	0.385	0.412	0.423	0.420
Weather → ETTm1	0.350	0.383	0.411	0.423	0.368	0.392	-	-	-	-	0.382	0.403	0.382	0.395
Avg	0.392	0.413	0.429	0.434	0.422	0.428	0.436	0.460	0.780	0.693	0.458	0.451	0.535	0.488
$14.7\%$  MSE reduction and  $12.0\%$  MAE reduction compared to the contrastive baseline CoST [46]. It is also notable that both Ti-MAE [21] and TST [56] outperform all the contrastive-based baselines. This indicates that masked modeling based on point-wise reconstruction suits the forecasting task better than the series-wise contrastive pre-training.
Cross-domain. As demonstrated in Table 3, we present multiple scenarios to verify the effectiveness under the cross-domain setting, where SimMTM consistently outperforms other baselines. Note that the channel-independent encoder enables the comparing baselines to be capable of transferring pre-trained models between datasets with different variate numbers: Weather  $\rightarrow$  {ETTh1, ETTm1}. But for LaST and TF-C with model-specific designs, they cannot be applied to these scenarios. While negative migration has been observed in some cross-domain scenarios, such as Weather  $\rightarrow$  ETTh1 and ETTh2  $\rightarrow$  ETTm1, SimMTM is still significantly overall superior to other baselines.
Table 4: In- and cross-domain settings of classification. For the in-domain setting, we pre-train and fine-tune the model on the same dataset Epilepsy. For the cross-domain setting, we pre-train a model on SleepEEG and fine-tune it to multiple target datasets: Epilepsy, FD-B, Gesture, EMG. Accuracy  $(\%)$  score are recorded. Full results are included in Appendix E.
Models	SimMTM	Random init.	Ti-MAE [21]	TST [56]	LaST [42]	TF-C [57]	CoST [46]	TS2Vec [55]
Epilepsy → Epilepsy	94.75	89.83	80.34	80.89	92.11	93.96	92.35	92.33
SleepEEG → Epilepsy	95.49	89.83	73.45	82.89	86.46	94.95	93.66	94.46
SleepEEG → FD-B	69.40	47.36	67.98	65.57	46.67	69.38	54.82	60.74
SleepEEG → Gesture	80.00	42.19	75.54	75.12	64.17	76.42	73.33	73.33
SleepEEG → EMG	97.56	77.80	63.52	75.89	66.34	81.74	73.17	80.92
Avg	87.44	69.40	72.17	76.07	71.15	83.29	77.47	80.36
Figure 4: Ablations of SimMTM on the reconstruction loss  $(\mathcal{L}_{\mathrm{rec.}})$  and constraint loss  $(\mathcal{L}_{\mathrm{con.}})$  in time series forecasting (left part) and classification (right part) tasks under both in-domain and cross-domain settings. More ablations are included in Appendix E.
4.3 Classification
In-domain. We investigate the in-domain pre-training effect on the classification tasks in Table 4. Note that different from forecasting, the classification task requires the model to learn high-level time series representations. From Table 4, we can find that the contrastive pre-training baselines achieve competitive performances. In contrast, the masking-based model Ti-MAE [21] and TST [56] perform poorly, and TST even exhibits a negative transfer phenomenon compared to the random initialization, indicating that contrastive learning is generally more suitable for classification tasks. Surprisingly, although SimMTM follows the masked modeling paradigm, with our specially-designed reconstruction task, it can still achieve the best performance in the classification task. This is benefited from the neighborhood aggregation from multiple masked series, which enables the model to exploit the local structure of the time series manifold.
Cross-domain. We experiment with four cross-domain transfer scenarios in Table 4: SleepEEG  $\rightarrow$  {Epilepsy, FD-B, Gesture, EMG}, where the target datasets are distinct from the pre-training dataset. Due to the large gap between pre-training and fine-tuning datasets, the baselines perform poorly in most cases, while SimMTM still surpasses other baselines and the random initialization significantly. Especially for SleepEEG  $\rightarrow$  EMG, SimMTM remarkably surpasses previous state-of-the-art TF-C (Accuracy:  $97.56\%$  vs.  $81.74\%$ ). These results demonstrate that SimMTM can precisely capture valuable knowledge from pre-training datasets and uniformly benefit extensive downstream datasets.
Ablations. As shown in Figure 4, we provide ablations to two parts of the training loss in SimMTM. It is observed that both  $\mathcal{L}_{\mathrm{reconstruction}}$  and  $\mathcal{L}_{\mathrm{constraint}}$  are essential to the final performance. Especially, for the SleepEEG  $\rightarrow$  EMG experiment, SimMTM surpasses the random initialization remarkably, where reconstruction and constraint losses provide  $9.7\%$  and  $16.0\%$  absolute improvement respectively. Besides, we can also find that in comparison to  $\mathcal{L}_{\mathrm{reconstruction}}$ ,  $\mathcal{L}_{\mathrm{constraint}}$  provides more contributions to the final results. This comes from our design that the constraint loss uncovers a proper time series
Table 5: Representation analysis for different methods in classification and forecasting tasks. For each model, we calculate the Centered Kernel Alignment (CKA) similarity  $(\%)$  [18] between representations from the first and the last encoder layers to measure the representation-learning property of deep models. Since the bottom layer representations usually contain low-level or detailed information, a smaller CKA similarity means the top layer includes different information from the bottom layer and indicates the model tends to learn high-level representations or more abstract information. For comparison, we also calculate the  $|\Delta_{\mathrm{CKA}}|$  between pre-trained and fine-tuned models, where a smaller value indicates a smaller representation gap between pre-training and fine-tuning, and the representations have stronger universality and portability.
Tasks	Ti-MAE [21]	TST [56]	LaST [42]	TF-C [57]	CoST [46]	TS2vec [55]	SimMTM
Classification	Pre-training	84.12	54.98	82.01	85.78	60.74	70.01	33.87
Fine-tuning	87.26	55.80	79.56	86.30	62.24	69.79	32.84
|ΔCKA|	3.14	0.82	2.45	1.53	1.50	0.22	1.04
Forecasting	Pre-training	83.60	99.76	75.20	59.35	87.09	70.20	97.79
Fine-tuning	91.24	94.92	79.25	60.60	77.38	83.73	97.89
|ΔCKA|	7.64	4.84	4.05	1.25	9.70	13.53	0.11
Sum|ΔCKA|	10.78	5.66	6.50	2.77	11.20	13.75	1.15
Table 6: Performance by applying SimMTM to four advanced time series forecasting models under the in-domain setting. “+ Sub-series Masking” refers to the sub-series masked modeling pre-training method proposed by PatchTST [26] itself. MSE and MAE are averaged from all prediction lengths in {96,192,336,720}. The detailed results for each forecasting horizon are in Appendix E.
Dataset	ETTh1	ETTh2	ETTm1	ETTm2
Model	MSE	MAE	MSE	MAE	MSE	MAE	MSE	MAE
Transformer [39]	1.088	0.836	4.103	1.612	0.901	0.704	1.624	0.901
+ SimMTM	0.927	0.761	3.498	1.487	0.809	0.663	1.322	0.808
Autoformer [47]	0.573	0.573	0.550	0.559	0.615	0.528	0.324	0.368
+ SimMTM	0.561	0.568	0.543	0.555	0.553	0.505	0.315	0.360
NS Transformer [24]	0.570	0.537	0.526	0.516	0.481	0.456	0.306	0.347
+ SimMTM	0.543	0.527	0.493	0.514	0.431	0.455	0.301	0.345
PatchTST [26]	0.417	0.431	0.331	0.379	0.352	0.382	0.258	0.317
+ Sub-series Masking	0.430↓	0.445↓	0.355↓	0.394↓	0.341	0.379	0.258	0.318↓
+ SimMTM	0.409	0.428	0.329	0.379	0.348	0.378	0.254	0.313
manifold helpful for reconstruction from multiple masked series, without which the neighborhood aggregation will degenerate to the trivial average.
Representation analysis. To illustrate the advantages of SimMTM intuitively, we provide a representation analysis in Table 5, where we can find the following observations. Firstly, we can find that the CKA value of SimMTM in the classification task is clearly smaller than the values in the forecasting task, where the former is a high-level task and the latter requires low-level representations. These results demonstrate that SimMTM can learn adaptive representations for different tasks, which can be benefited from our design in the pre-training loss. Concretely, the temporal variations of classification pre-training datasets are much more diverse than the forecasting datasets. Thus, the  $\mathcal{L}_{\mathrm{constraint}}$  will be easier for optimization in classification, deriving a smaller CKA value. Secondly, from  $|\Delta_{\mathrm{CKA}}|$ , it is observed that the models pre-trained from SimMTM present a smaller gap with respect to the final fine-tuned model in representation learning properties, which is why SimMTM can consistently improve downstream tasks.
Model generality. From Table 6, we can find that as a general time series pre-training framework, SimMTM can consistently improve the forecasting performance of diverse advanced base models, even for the state-of-the-art time series forecasting model PatchTST [26]. This generality also
Figure 5: Model analysis. Left part is for fine-tuning ETTh2 pre-trained model to ETTh1 with limited data, where a smaller MSE indicates better performance. Right part presents the MSE performance of SimMTM in the ETTh1 "input-336-predict-96" in-domain setting with different masked ratio  $r$  and numbers of masked series  $M$ , where a darker red means better performance.
indicates that we can further improve the model's performance by employing advanced base models as encoders. It is also notable that different from the negative transfer phenomenon caused by the canonical sub-series masked modeling used in the PatchTST paper [26], the consistency improvement of SimMTM further verifies the effectiveness of our design.
Fine-tuning to limited data scenarios. One essential application of pre-training models is to provide prior knowledge for downstream tasks, especially for limited data scenarios, which is critical to the fast-adaption of deep models. Thus, to verify the effectiveness of SimMTM and other pretraining methods in data-limited scenarios, we pre-train a model on ETTh2 and fine-tune it to ETTh1 with different choices for the remaining proportions of training data. All results are presented in Figure 5. We can find that SimMTM achieves significant performance gains in different data proportions compared to other time series pre-training methods. Specifically, for the  $10\%$  data fine-tuning setting, SimMTM significantly outperforms the advanced masking-based method Ti-MAE [21] (MSE: 0.591 vs. 0.660). Compared with the contrastive-based method TF-C [57], SimMTM also achieves  $26.0\%$  MSE reduction. These results further verify that SimMTM can effectively capture valuable knowledge from datasets and boost the final performance, even in limited data scenarios.
Masking strategy. Note that the difficulty of reconstructing the original time series increases along with the increase of the masked ratio, but decreases when the number of neighbor masked series increases. We explore the potential relationship between the masked ratio and the number of masked series used for reconstruction, namely  $r$  and  $M$  in Eq. (1) respectively. The experimental results in Figure 5 show that we need to set  $M \propto r$  to obtain better results. Experimentally, we choose the masking ratio as  $50\%$  and adopt three masked series for reconstruction throughout this paper. In addition, we can find that only using one masked series ( $M = 1$ ) for pre-training generally performs worse than the settings with larger  $M$ , where the latter enables the model to discover the relations between input series and its neighbors. See Figure 1 for an intuitive understanding. These results further highlight the advantage of our design in manifold learning.
This paper presents SimMTM, a simple pre-training framework for masked time-series modeling. Going beyond the previous convention in reconstructing the original time series from unmasked time points, SimMTM proposes a new masked modeling task as reconstructing the original series from its multiple neighbor masked series. Concretely, SimMTM aggregates the point-wise representations based on the series-wise similarities, which are carefully constrained by the neighborhood assumption on the time series manifold. Experimentally, SimMTM can furthest bridge the gap between pretrained and fine-tuned models, thereby achieving consistent state-of-the-art in distinct forecasting and classification tasks compared to the most advanced time series pre-training methods, covering both in-domain and cross-domain settings. In the future, we will further extend SimMTM to large-scale and diverse pre-training datasets in pursuing the foundation model for time series analysis.
This work was supported by the National Key Research and Development Plan (2021YFB1715200), National Natural Science Foundation of China (62022050 and 62021002), and Beijing Nova Program (Z201100006820041).
[42] Zhiyuan Wang, Xovee Xu, Weifeng Zhang, Goce Trajcevski, Ting Zhong, and Fan Zhou. Learning latent seasonal-trend representations for time series forecasting. In NeurIPS, 2022.
[43] Wetterstation. Weather Dataset. https://www.bgc-jena.mpg.de/wetter/.
[44] Alexander Wettig, Tianyu Gao, Zexuan Zhong, and Danqi Chen. Should you mask  $15\%$  in masked language modeling? In EACL, 2023.
[45] Kristoffer Wickström, Michael Kampffmeyer, Karl Øyvind Mikalsen, and Robert Jenssen. Mixing up contrastive learning: Self-supervised representation learning for time series. PRL, 2022.
[46] Gerald Woo, Chenghao Liu, Doyen Sahoo, Akshit Kumar, and Steven Hoi. Cost: Contrastive learning of disentangled seasonal-trend representations for time series forecasting. In ICLR, 2022.
[47] Haixu Wu, Jiehui Xu, Jianmin Wang, and Mingsheng Long. Autoformer: Decomposition transformers with auto-correlation for long-term series forecasting. In NeurIPS, 2021.
[48] Zhirong Wu, Yuanjun Xiong, Stella X. Yu, and Dahua Lin. Unsupervised feature learning via non-parametric instance discrimination. In CVPR, 2018.
[49] Zhenda Xie, Zigang Geng, Jingcheng Hu, Zheng Zhang, Han Hu, and Yue Cao. Revealing the dark secrets of masked image modeling. In CVPR, 2023.
[50] Zhenda Xie, Zheng Zhang, Yue Cao, Yutong Lin, Jianmin Bao, Zhuliang Yao, Qi Dai, and Han Hu. Simmim: A simple framework for masked image modeling. In CVPR, 2022.
[51] Jiehui Xu, Haixu Wu, Jianmin Wang, and Mingsheng Long. Anomaly transformer: Time series anomaly detection with association discrepancy. In ICLR, 2021.
[52] Ling Yang and Shenda Hong. Unsupervised time-series representation learning with iterative bilinear temporal-spectral fusion. In ICML, 2022.
[53] Xinyu Yang, Zhenguo Zhang, and Rongyi Cui. Timecr: A self-supervised contrastive learning framework for univariate time series representation. KBS, 2022.
[54] Hugo Yèche, Gideon Dresdner, Francesco Locatello, Matthias Hüser, and Gunnar Rätsch. Neighborhood contrastive learning applied to online patient monitoring. In ICML, 2021.
[55] Zhihan Yue, Yujing Wang, Juanyong Duan, Tianmeng Yang, Congrui Huang, Yunhai Tong, and Bixiong Xu. TS2Vec: Towards Universal Representation of Time Series. In AAAI, 2022.
[56] George Zerveas, Srideepika Jayaraman, Dhaval Patel, Anuradha Bhamidipaty, and Carsten Eickhoff. A transformer-based framework for multivariate time series representation learning. In SIGKDD, 2021.
[57] Xiang Zhang, Ziyuan Zhao, Theodoros Tsiligkaridis, and Marinka Zitnik. Self-supervised contrastive pre-training for time series via time-frequency consistency. In NeurIPS, 2022.
[58] Haoyi Zhou, Shanghang Zhang, Jieqi Peng, Shuai Zhang, Jianxin Li, Hui Xiong, and Wancai Zhang. Informer: Beyond efficient transformer for long sequence time-series forecasting. In AAAI, 2021.
All the experiments are repeated five times, implemented in PyTorch [28] and conducted on NVIDIA A100 SXM4 40GB GPU. We implement the baselines based on their official implementation and follow the configuration from their original papers. For the metrics, we adopt the mean square error (MSE) and mean absolute error (MAE) for the time series forecasting. As for the classification, accuracy, precision, recall, F1 score, and their average value are recorded.
A.1 Dataset Description
We conduct experiments to evaluate the effect of our method under in-domain and cross-domain settings on twelve real-world datasets for two typical time series analysis tasks: forecasting and classification, covering diverse application scenarios (electricity system, neurological healthcare, human activity recognition, mechanical fault detection, and physical status monitoring), different types of signals (ECG, EMG, acceleration, vibration, power load, weather, and transoirtation), multivariate channel dimensions (from 1 to 862), varying times series lengths (from 96 to 5120) and large span sampling ratio (from  $100\mathrm{Hz}$  to  $4000\mathrm{Hz}$ ). The detailed descriptions of these datasets are summarized in Table 7.
Table 7: Dataset descriptions. Samples are organized in (Train/Validation/Test).
Tasks	Datasets	Channels	Length	Samples	Classes	Information	Frequency
Forecasting	ETTh1,ETTh2	7	{96,192,336,720}	8545/2881/2881	-	Electricity	1 Hour
ETTm1,ETTm2	7	{96,192,336,720}	34465/11521/11521	-	Electricity	15 Mins
Weather	21	{96,192,336,720}	36792/5271/10540	-	Weather	10 Mins
Electricity	321	{96,192,336,720}	18317/2633/5261	-	Electricity	1 Hour
Traffic	862	{96,192,336,720}	12185/1757/3509	-	Transportation	1 Hour
Classification	SleepEEG	1	200	371005/-/-	5	EEG	100 Hz
Epilepsy	1	178	60/20/11420	2	EEG	174 Hz
FD-B	1	5120	60/21/135599	3	Faulty Detection	64K Hz
Gesture	3	315	320/120/120	8	Hand Movement	100 Hz
EMG	1	1500	122/41/41	3	Muscle responses	4K Hz
(1) ETT (4 subsets) [58] contains the time series of oil temperature and power load collected by electricity transformers from July 2016 to July 2018. ETT is a group of four subsets with different recorded frequencies: ETTh1/ETTh2 are recorded every hour, and ETTm1/ETTm2 are recorded every 15 minutes.
(2) WEATHER [43] includes meteorological time series with 21 weather indicators collected every 10 minutes from the Weather Station of the Max Planck Biogeochemistry Institute in 2020.
(3) ELECTRICITY [38] records the hourly electricity consumption of 321 clients from 2012 to 2014. Values are in kW of each  $15\mathrm{min}$ . All time labels report to Portuguese hour. However, all days present 96 measures  $(24\times 4)$ . For every year in March, time change day (which has only 23 hours), values between 1:00 am and 2:00 am are zero for all points. For every year in October, time change day (which has 25 hours), the values between 1:00 am and 2:00 am aggregated consumption of two hours.
(4) TRAFFIC [29] encompasses the hourly measures of road occupancy rates obtained from 862 sensors situated in the San Francisco Bay area freeways. These measurements were carried out between January 2015 and December 2016.
(5) SLEEPEEG [16] contains 153 whole-night sleeping electroencephalography (EEG) recordings from 82 healthy subjects. We follow the same data preprocessing approach as [57] and get 371,055 univariate brainwaves. Each brainwave is sampled at a frequency of  $100\mathrm{Hz}$  and associated with one of five sleeping stages: Wake, Non-rapid eye movement (3 sub-states), and Rapid Eye Movement.
(6) EPILEPSY [1] monitors the brain activities of 500 subjects with a single-channel EEG sensor. Every subject is recorded for 23.6 seconds of brain activities. The dataset is sampled at  $178\mathrm{Hz}$  and contains 11,500 samples in total. We follow the procedure described by [57]. The first four classes (eyes open, eyes closed, EEG measured in the healthy brain region, and EEG measured in the tumor region) of the original five categories of each sample are classified as positive, and the remaining classes are used as negative.
(7) FD-B [19] is generated by electromechanical drive systems. It monitors the condition of rolling bearings and detects their failures based on the monitoring conditions, which include speed, load torque, and radial force. Concretely, FD-B has 13,640 samples in total. Each recording is sampled at  $64\mathrm{kHz}$  with 3-class labels: undamaged, inner damaged, and outer damaged.
(8) GESTURE [22] are collected from 8 hand gestures based on the paths of hand movement recorded by an accelerometer. The eight gestures are hand swiping left, right, up, and down, hand waving in a counterclockwise or clockwise circle, hand waving in a square, and waving a right arrow, respectively. This dataset contains 440 examples of balanced classification labels that can be used, and each sample includes eight different kinds of gesture categories.
(9) EMG [30] is sampled with  $4\mathrm{KHz}$  and consists of 163 single-channel EMG recordings from the anterior tibialis muscle of three healthy volunteers suffering from neuropathy and myopathy. Each patient is a classification category, so each sample is associated with one of three classes.
A.2 Baselines Implementation
We compare SimMTM against six state-of-the-art baselines. To make a fair and comprehensive comparison, we tried two baseline implementation approaches for forecasting and classification tasks: the unified encoder and reproduced with their official implementation encoder. Notably, LaST [42] and TF-C [57] are closely related to model structures. We directly report results from their papers or reproduce codes with official implementation.
Table 8: Baselines implementation details.
Baselines	Task	Encoder	Performance Comparison	Report
Ti-MAE [21]	Forecasting	Channel-independent Transformer	better	Main text
Official implementation	Section E
Classification	1D-ResNet	better	Main text
Official implementation	Section E
TST [56]	Forecasting	Channel-independent Transformer	better	Main text
Official implementation	Section E
Classification	1D-ResNet	better	Main text
Official implementation	Section E
LaST [42]	Forecasting	Official implementation	/	Main text
Classification	Official implementation	/	Main text
TF-C [57]	Forecasting	Official implementation	/	Main text
Classification	Official implementation	/	Main text
CoST [46]	Forecasting	Channel-independent Transformer	better	Main text
Official implementation	Section E
Classification	1D-ResNet	better	Main text
Official implementation	Section E
TS2Vec [55]	Forecasting	Channel-independent Transformer	better	Main text
Official implementation	Section E
Classification	1D-ResNet	better	Main text
Official implementation	Section E