PowerPM: Foundation Model for Power Systems
Authors: Shihao Tu
Yupeng Zhang
Jing Zhang
Zhendong Fu
Yin Zhang
Yang Yang
Venue: NeurIPS 2024 poster (2024)
Keywords: time series pre-training
power systems
foundation model
Primary area: machine_learning_for_other_sciences_and_fields
TL;DR: PowerPM is a foundation model for power systems.

Abstract: The proliferation of abundant electricity time series (ETS) data presents numerous opportunities for various applications within power systems, including demand-side management, grid stability, and consumer behavior analysis. Deep learning models have advanced ETS modeling by effectively capturing sequence dependence. However, learning a generic representation of ETS data for various applications is challenging due to the inherently complex hierarchical structure of ETS data. Moreover, ETS data exhibits intricate temporal dependencies and is susceptible to the influence of exogenous variables. Furthermore, different instances exhibit diverse electricity consumption behavior. In this paper, we propose a foundation model PowerPM for ETS data, providing a large-scale, off-the-shelf model for power systems. PowerPM consists of a temporal encoder and a hierarchical encoder. The temporal encoder captures temporal dependencies within ETS data, taking into account exogenous variables. The hierarchical encoder models correlations between different levels of hierarchy. Furthermore, PowerPM leverages a novel self-supervised pre-training framework consisting of masked ETS modeling and dual-view contrastive learning. This framework enables PowerPM to capture temporal dependency within ETS windows and aware the discrepancy across ETS windows, providing two different perspectives to learn generic representation. Our experiments span five real-world scenario datasets, including both private and public data. Through pre-training on massive ETS data, PowerPM achieves SOTA
performance on diverse downstream tasks within the private dataset. Notably, when transferred to public datasets, PowerPM retains its edge, showcasing its remarkable generalization ability across various tasks and domains. Moreover, ablation studies and few-shot experiments further substantiate the effectiveness of our model.

PowerPM: Foundation Model for Power Systems
Shihao Tu Zhejiang University shihao. tužzju edu.
Yupeng Zhang' Zhejiang University yuppzhang@zju ju edu.
Jing Zhang Renmin University China thang-jing@inn edu. cn
Zhendong Fu Zhejiang University zhendongfugzju edu.
Yin Zhang
Yang Yang!
Zhejiang University yinzh@zju. edu.
Zhejiang University yangyašzju. edu.
Abstract
The proliferation of abundant electricity time series (ETS) data presents numerous opportunities for various applications within power systems, including demand-side management, grid stability, and consumer behavior analysis. Deep learning mod- els have advanced ETS modeling by effectively capturing sequence dependence. However, learning a generic representation of ETS data for various applications is challenging due the inherently complex hierarchical structure of ETS data. Moreover, ETS data exhibits intricate temporal dependencies and is susceptible the influence of exogenous variables. Furthermore, different instances exhibit diverse electricity consumption behavior. In this paper, we propose a foundation model PowerPM for ETS data, providing large-scale, off-the-shelf model for power systems. PowerPM consists a temporal encoder and a hierarchical encoder. The temporal encoder captures temporal dependencies within ETS data, taking into account exogenous variables. The hierarchical encoder models correlations between different levels of hierarchy. Furthermore, PowerPM leverages a novel self-supervised pre-training framework consisting of masked ETS modeling and dual-view contrastive learning. This framework enables PowerPM capture temporal dependency within ETS windows and aware the discrepancy across ETS windows, providing two different perspectives to learn generic representation. Our experiments span five real-world scenario datasets, including both private and public data. Through pre-training on massive ETS data, PowerPM achieves SOTA performance on diverse downstream tasks within the private dataset. Notably, when transferred to public datasets, PowerPM retains its edge, showcasing its remarkable generalization ability across various tasks and domains. Moreover, ablation studies and few-shot experiments further substantiate effectiveness of our model.
Introduction
The volume Electricity Time Series (ETS) data has recently increased rapidly due to the emergence of advanced power systems known smart grids [10). This abundance of data has paved the way for
diverse applications in power systems, including demand-side management [22], grid stability [2 and consumer behavior analysis [49, etc. Meanwhile, these applications have spawned various tasks, as shown in Fig. dd). These include load forecasting [27 4]. clock anomaly detection [46), electricity
theft 15 and and the detection of elderly individuals living alone [45).
'These authors contributed equally this work.
Conresponding authors.
38th Conference on Neural Information Processing Systems (NcurIPS 2024).

---

Demand-side management
-Load frecassing
Distrrki
Soli geseration forecaritg
Grid stabilitt
-Fleemeety ther detecticn
Consumer behavior analysis
Elderty living alone desection -High-pows appliacces dinection
dinteraction & cllsstnnce
vuriabees
08390 06000
12:00
18:00
24:00
... Total 44 masks
(a)
(b)
(c)
(d)
(a) The hicrarchical structure ETS data. (h) The temporal dependency within ETS data and the influence of exogenous variables. (c) Different electricity consumption behaviors exist across time and instances. (d) Various tasks in power systems.
Figure 1:
As society progresses towards modernization, electricity consumption is rapidly increasing, pre- senting opportunities and challenges for the development and application of smart grids. On one hand, the substantial economic benefits that accompany this significant electricity usage are con- siderable. On the other hand, unreasonable electricity planning can have a detrimental impact on the environment 30) Therefore, given the large volume of data and the variety of tasks, there is an urgent need to study effective ETS data modeling methods for these tasks, so to improve economic efficiency while adhering to low-carbon principles.
Recently, numerous research studies on pre- pre-training approaches for ETS data have emerged. These approaches adopt the "pre-training then fine-tuning" paradigm to deal with the dilemma of limited annotation data, and the pre-trained model to easily adapt to new tasks, such as PatchTST (21), TS2Vec [42]. CoST (37), etc. However, these pre- pre-training methods only utilize small-scale of data with small number of instances (e.g. users), resulting in poor performance on downstream tasks. As the same time, many researcher begin to apply Large Language Models (LLMs) to assist time series modeling by using pre-trained LLM to encode time series [5]] incorporating additional descriptions related to the time series 17 20 Nevertheless, these models have limited ability in the power system scenario due insufficient pre-training data of power systems and the lack of sufficient Nomain-speciff knowledge. Additionally, none of these models are tailored for the scenario of power systems, so they neglect the unique characteristics of ETS data. Consequently, there remains significant research gap in existing power systems literature regarding the modeling of ETS data using a foundation model.
In our scenario, the ETS data contains numerous instances and naturally exhibits a complex hierar- chy (3]. As depicted in Fig. [a), a city ETS can be disaggregated into district ETS accroding to the administrative divisions, which can further be disaggregated into user ETS in this district. For the complex hierarchy of ETS data, modeling ETS data entails the consideration of several challenges:
(1) Hierarchical Dependency Modeling. The hicrarchy of ETS data facilitates information inter- action across different granularities. Fine-grained ETS provides detailed insights into individual electricity usage, while coarse-grasd searse-grained ETS for districts and cities captures broader factors and indicates overall trends. For example, user-level data reflects user-specific behaviors and city-level data encompasses demographics and policy effects (29 35). Integrating these levels of granularity to provide both macro and micro perspectives is complex task that requires sophisticated modeling.
(2) Temporal Dependencies within ETS Window. An ETS window refers to a piece of electricity time series over a period of time. The temporal dependencies within an ETS window refer to the correlations and dependencies between observations at different timestamps. As shown in Fig, [bb, the city-level ETS exhibits daily and weekly dependency. Moreover, the temporal dependencies are often influenced by exogenous variables, such as weather, temperature, and seasonal effects. Integrating these factors into the model is challenging because their impact may interact with the temporal dynamics in complex ways. Accurately capturing the temporal dependencies with the impact of exogenous variables is a key challenge in modeling ETS data.
(3) Discrepancy across ETS Windows. The patterns observed ETS windows can vary significantly across different instances and different timestamps. For instance, as shown in Fig. [c), residential electricity consumption (User A) reaches its peak in the mornings and evenings, used for lighting, appliances, and heating, However, electricity usage typically declines during the day because residents

---

Demand-side Maragement
Grid Stability
liehavior Analysis Detecton (1.3)
Impurttion (45E)
Coafii
One Iay
0.
Towethi
Impulation 205
Clock Dinection (R2.5
Forcusting MABE
SwenerggM
High poweyapsiir
Santh Stuucure
Induyyy
Elaghiity Theth Detection (F0.3
Kiy Towcceting/M331
Gerter
Impone on dhbs) (b)
Class Featoor (Acc)
(a)
Figure 2: Performance comparison our model and other baseline models on all downstream tasks in our scenario. Model performances are plotted on radar subfigures for clarity with the same coordinate range.
are generally absent, being engaged in work or education activities outside the home. Moreover, industries (User B) have high power demand during specific daytime periods for machinery and production lines, with lower load requirements during nighttime and weekends. These variations in behavior highlight the challenge of achieving consistency across ETS windows in personalized modeling.
To address these challenges, we propose foundation model for power systems named Power Pre- trained Model (PowerPM), as illustrated in Figure 3 PowerPM contains about 250M parameters and is pre-trained on large-scale hierarchical ETS data with 987.42GB. Specifically, we employ the "pre- training then fine-tumin tiee-tuning" paradigm to learn generic representations by pre-training on hierarchical ETS data and to unify various tasks by fine-tuning downstream data. During pre-training stage, we propose a novel self-supervised pre- pre-training framework consisting of masked ETS modeling and dual-view contrastive learning, which enables PowerPM to capture temporal dependency within ETS windows and aware the discrepancy across ETS windows, so as to provide two different perspectives to learn universal representations. PowerPM mainly consists of two modules, namely, temporal encoder and hierarchical encoder. The temporal encoder employs Transformer encoders to capture the temporal dependency in ETS data, and incorporates exogenous variables make the modeling process more robust. Moreover, to model hierarchical dependency, hierarchical encoder utilizes R-GCN (55 to propagate information about the correlation between hierarchy. According the message that passes through the hierarchies, the micro and macro information can effectively assist in modeling the ETS data. summary, the main contributions of our work include:
1. We propose foundation model for power systems named PowerPM, which is pre-trained on large-scale ETS data and provide an off-the-shelf model for power systems.
2. To the best of our knowledge, PowerPM is the first date that considers temporal de- pendency and hierarchical dependency simultaneously. In addition, we present novel self-supervised pre-training framework that combines masked ETS modeling and dual-view contrastive learning, enhancing the model's 's ability to learn temporal dependencies within ETS windows and aware the discrepancy across ETS windows.
3. Extensive experiments show that PowerPM generalizes well to 44 downstream tasks. Fig. summarizes the results of all the downstream tasks, showing its great potential in ETS data modeling. Moreover, when transferred to the public dataset, PowerPM maintains its superiority, showcasing its remarkable generalization ability across various tasks and domains. Further analysis illustrates the effectiveness of PowerPM as well.
2 Methodology
Overview. As shown in the middle part of Fig. 3 Firstly, the hierarchical graph G is constructed according to the naturally existing hierarchical relationship of ETS data. The ETS windows in & and its corresponding exogenous variables are denoted as (x, N and { where N is the number of instances, x; E RTw,. O, E x and each instance ETS window spans Tú time points starting at T, and ending at Tj. Each time point has K kinds of cxogenous variables. Our objective is to perform
N
3

---

Laver + LMSE
R-GEN
Transformer Encoder
Embedding Tahle
25
Hierarchical Encoder
fn
h
Diantrt-Citt
2;
Projectins
e
My.
Clater-uarict
Adjacennc Marris
Temporal Encoder
fr
Pst
Indexing
CitySharic
Exegrentt Vartalkee
tP.
Patched Window
Exegeucus Yariables
Smucture
Herarrciicl Groph G
Self-supervised Pre-training
modeling
Cunstrsciive Learning
Parching
Peronsrucy
12
1c
22
Hierarchical Graph Construction
prodict
Poditio Pair Terpprs Viea
Naaerive Pax is Taspocall Niew
City
Claster
Raadon Cosa:
Nogrive Instacc
The pre- treiniinn framework of PowerPM. For simplicity, we take the windows of each instance in the same time range for illustration, and the window process at other times is the same.
Figure 3:
pre-training encoder f()) to encode each window into latent representation RR-xd where d indicates the dimension of the latent representation. More specific, PowerPM consists of exogenous variable enhanced temporal encoder fr(+) and hierarchical encoder f-),,, with the process: z; f(,,, o,, () fH (fr o,), g). In addition, a novel self-supervised strategy which combines masked ETS modeling and duall-iew contrastive learning is used for pre-traine pre-training PowerPM. Next, we will detail the techniques both model architecture and pre-training strategy.
2.1 Hierarchical Graph Construction
The data of cities, districts, and users in ETS data naturally form a hierarchical relationship. based on which we can construct a hicrarchical graph. However, the imbalance in the number of users and districts means there will be multitude of edges between nodes and district nodes, which significantly increases the complexity of graph modeling. To address this, we employ a clustering strategy create intermediary nodes, which is common approach implement graph sparsifica- tion [13] and a user group policy in the power systems [36 44 12]. As depicted in Fig. (c), we clustering method to categorize users into several clusters, the detailed process
be found in App. B.I The cities are bidirectionally connected to districts, and these user clusters are also bidirectionally connected districts but unidirectionally connected to districts. By sparsifying the edges, we enhance the efficiency of graph modeling. Mathematically, we represent the hierarchy as directed graph G (V, R). where is the set of nodes, each node corresponds to an instance, E is the set of directed edges, and R is the set of type of edges (e.g. user cluster - district, district user, etc.).
2.2 Temporal Encoder with Exogenous Variables
Patching, In the G, each node's feature æ; is a window of ETS data corresponding to instance i. Due to the semantic sparsity of time series, patch each window æ; into N, segments, each of length P. resulting in P, IR N where N₂ Tw Tu-PP P1 and this method proved its validity in s many works (2 17 20 Subsequently, linear projection is applied to each segment to obtain the window representation h; Ny
Exogenous Variables Encoding. To efficiently interact with exogenous variables, we model these variables using learnable embeddings E E REX M,)xd where K indicates the number of exogenous variables (e.g. weather type and temperature). Mr represents the number of value types of the k-th exogenous variable (e.g. sunny and rainy in weather type variable). The exogenous variables of E N corresponding to p, of the k-th exogenous variable are used to obtain representations of the exogenous variables from E, indexing out e N, illustrated in

---

Fig. (b). Subsequently, we derive representation u, E IR Nydd that considers the window's exogenous variable influence: u; h, > k
(A)
Temporal Encoder. To model the complex temporal dependency and interaction with exogenous variables, we use the vanilla Transformer encoder [34 to encode u, resulting in augmented temporal representation 2, E RN
2.3 Hierarchical Encoder
To model the complex correlation across different hierarchies, we employ Graph Neural Networks (GNNs). GNNs have recently become increasingly popular for modeling relationships within time addition, considering that the correlation relationships of different edges are distinct, we adopt R-GCN [25] to integrate information across various hierarchies and instances, as depicted in (a). Specifically, we use R-GCN to update the representation by considering its neighboring nodes in G, with the final node representation denoted as z; E R N xd Moreover, we use z; to perform self-supervised pre-training.
series data, which enhances temporal representation [ 26 40..
2.4
Sell-supervised Pre-training
2.4.1 Masked ETS Modeling
To model temporal dependency within an ETS window, we have adopted the widely utilized masked reconstruction strategy. Nevertheless, existing random masking methods may face a significant challenge: they reconstruct the missing part based on the known surrounding part [21 8), without considering the prediction of future parts relying solely the past part. This approach not only diminishes the difficulty of the pre- pre-training stage but also lacks consistency across pre-training task and forecasting task.
To address this issue, we propose a novel masking approach that combines random and casual masking, as shown in Fig. (d) (left). Specifically, we randomly select one of the masking approaches for given patched window p:. resulting masked p.. This approach not only retains the benefits of the random masking strategy but also ensures that the model learns to predict future parts based solely on past information, thereby can more comprehensively capture the temporal dependencies within
Mask, if < 0.5 a window. Mathematically, this can be formulated as: masked p: Mask.(p.) otherwise
where Mask, and Mask, denote the random and causal masking, respectively, and a E [0. 1] is uniformly distributed variable. Specifically, after the æ; is inputted into PowerPM for masked ETS modeling, we will obtain a reconstructed é. The corresponding reconstruction loss is: LMSE = (æ. - 2.
1
N
2.4.2 Dual-view Contrastive Learning
The objective of contrastive learning is to learn representations by bringing positive pairs closer and pushing negative pairs farther apart in the latent space [5] G) Motivated by this, make PowerPM aware of the discrepancy across ETS windows, we employ dual-view contrastive learning (DVCL) to discern subtle differences in electricity usage behavior.
Positive and Negative Sample Pairs. These pairs are determined from two views: one is temporal view, which is based on the time difference between the two windows. Another is the inslance view, which depends on whether two windows belong to the same instance. For the same instance, the closer the time difference between two windows, the closer their representations are likely to be. This idea is also presented in 31 42). Conversely, windows from different instances or the same instance with larger time difference are likely have more distinct representations. Overall, we consider adjacent windows from the same instance positive samples, while windows from different instances or non-adjacent windows from the same instance are negative samples. (d) (right), for the district node V in G, the original start timestamp about this window is Te. After shifting several time steps on, we obtain another window V+ starting at T, + 8, which serves as a positive sample. Meanwhile, we select windows from other nodes in G. such as city P, starting at Ta. as well as windows from the same node V but starting at Te, where TT Tal > 8. These windows serve instance and temporal negative samples, respectively, and are denoted as P and V-
depicted in Fig.
5

---

Mathematically, given an ETS window D₁, we obtain a positive sample x by shifting it by 8 time steps. The other samples this batch serve as negative samples, totaling B neg- ative samples, where B is the batch size during pre-training. The DVCL. loss is: LDVCL = (pp(sm([(#))))C) ()//)) E== log _ [-exp(simm( ((x)) (a))
where I is the boolean vector to select the negative pairs and sim(.) is cosine similarity function.
3 Experiments
3.1 Experiment Setup
Pre-training Dataset. PowerPM is pre-trained on mount of ETS data, private dataset from the real scenarid This pre-training dataset encompasses ETS data of cities, districts, and users, covering over 3 years records. The ETS data is collected frequency of one data point every 15 minutes. More details are in App. A
Downstream Dataset. To evaluate the performance of PowerPM, we conduct comprehensive experiments on eleven downstream private and public datasets. And seven private datasets are also collected from real scenario. These datasets have different labels for different tasks, Among them, the solar generation dataset does not have a hierarchical structure due to its particularity. Four public datasets are obtained from CSISO ISONE NYISO and PJM and they all exhibit a hierarchical structure. Further details can be found in Appendix A
Settings. For the model configurations, the temporal encoder contains 26-layer Transformer encoder with model dimension 1024, inner dimension (FFN) 2048 and 16 attention heads, and the hierarchical encoder contains 2-layer R-GCN. PowerPM contains about 250M parameters. During pre-training, the 40% segments in each input window are masked the form of random mask and casual mask, the user cluster numbers is set to 12. See further details in App. B.1
Baselines. We compare with state-of-the-an methods: Large Language Model (LLM) enhanced models: GPT4TS [5I. Time-LLM 17.. UniTime [20]; pre-train models: PatchTST 21, CoST 337.
TS2Vec 42]; supervised models: DLinear 43 TimesNet (88. More implementation details are B.2
provided in App.
Evaluation Metries. For forecasting and imputation tasks, we mean squared error (MSE): + (y- ")²² and mean absolute error (MAE): Ei= /y - 9 the evaluation metric. For classification tasks, we use accuracy as the metric. The metric of the anomaly detection task includes precision, recall, FO.5, and Fl scores. The F is metric defined the weighted harmonic mean
of precision and recall, with the following equation: F3 (1+8*) xprrecxsion exprecisionxreaatt
We use FO.5
xpxprccisnonn recall
for anomaly detection, since precision is more important than recall in power systems scenario [15]
3.2 Downstream Tasks
Demand-side Management. Demand-side management aims to optimize and balance the power system by managing and adjusting the electricity demand of end-users. We develop tasks to predict load at different levels (such as cities and users) and tasks to forecast solar generation. With demand- side management, we can better plan and schedule power resources, improve energy efficiency, promote the development of renewable energy, and achieve sustainable energy management.
Grid Stability. To ensure the stability of the power grid, have implemented a series of tasks, including electricity theft detection, load imputation, and clock anomaly detection, to address the impact of potential appliance failures within the grid and external electricity theft on the quality of power data and grid operations. Internal appliance malfunctions within the grid such as clock anomalies or the inability to record electricity usage accurately decrease the accuracy of power data, making it challenging for power dispatch and management. Additionally, external electricity theft
Due privacy concerns of the dataset and the company, mask the specific information. httt:://ww.eeeerget iso http://www...reeeeeooc///e//easp https://dataminer2,,
and atd.///ddaddddddddddddada

---

can cause economic losses and pose threat to the stable operation and reliability of the power grid, potentially causing power outages and other adverse effects.
Consumer Behavior Analysis. To provide users with more assistance, we have implemented tasks such detection of elderly living alone, high-power appliance detection, gender classification, age classification, and family structure classification. Additionally, we can provide more flexible power scheduling plans for special groups, so as to optimize power dispatch. We also aim to understand the energy usage differences among different genders and age groups and provide personalized energy management recommendations and services for different users.
Table 1: Performance comparison on private dataset. The result of MAE metric refer to Tab. 6
Taaks
PowesPM
PowaPM
GPT4TS 5D
TacCIM
UniTime
PachTST
T52Ve
TleasNec
DLiscal
MSC
345B
MBB
MBE
MS2
305E
MS2
MSC
MBB
Rodtste User
Reecasting
258
96
1.4133
0.477)
01:311
0.2015
0.088
0.340
0:531
0.4812
0.5307
0.5370
41-2514
0. 0.9992
103313
0.537
0.3166
0.668
0.563
05585
0.5013
0.5670
05522
0.500
0.579
0.5015
0.4355
0.667
0,5436
0.2383
0.297
03344
10.353
02237
0,2111
04606
0.3506
0,3883
0.391
Ivnezsaing
671
745
0.3226
0.1818
10 866
10
0.475
04414
038375
0.355
03178
0.566
03306
0.6559
05571
0,5111
0.960
Avg
0.931
0.39M
0.191
8.467
0.4397
0.4244
0.2882
0.736
03339
20244
0.3155
0.549
03357
0.360
0,4135
0,5001
T myy
Feeccaling
Distit
285
06
0.2926
0.3000
03348
00310
03531
03336
0.3656
013454
0.3555
0.3005
0. 0.590
0.4455
0.416
0.4507
0.456
0.4742
0,4050
0.4413
0.5186
0.5690
03671
6.3499
035591
04473
04492
04664
02213
02754
0.3630
0.364
0.315
0.2000
0.5433
Reecedinng
City
672
768
0.3084
6.5211
6.7190
0.3015
N15727
0.5632
0.2266
0.3540
0.165
B46666
0L3996
Avg.
03193
0.3257
003109
03333
03419
0.415
03551
0,8993
01121
0.229
0.1315
01551
0.06/8
0.350
0.3579
GA177
Ivnerseiig
Gimerazze
06
0,1223
0.335
MIME
01194
0.2852
02113
0.3366
0.4688
10.776
103538
0.352
0.468
0.365
0.3506
04216
0,4710
40335
03881
0.300
0.610
0 0.775
03366
0.485
0.635
0.3443
Teshebe Dser
Imperation
0.375
0.25
0.5
0.3388
0.366
0.315
03571
03446
04341
0.005
0.443
0.350
04495
0.4550
0331
10,4000
1033155
033155
0.4105
0.4805
0.4150
0.455
0.5066
0.4275
0.510
0.6634
0.5355
Avg
03679
0.9922
0.125
0.134
102651
02997
0.3614
0,2977
0,3075
0.3516
0,3223
0,3006
0,3544
Inspatation
Public
0.75
01100
0.3776
40.599
2019
0.633
03111
0331
0.1776
0.4119
1.410
0.603
0.1866
044011
0.505
0,597
Ang
1100
40310
03013
0.311
0.7558
0.5816
041968
04114
0.590
0-434
Imeottatio
Distict
0,375
025
0.3666
0.1384
0 2200
0.1059
02223
03226
03505
0240
102381
011700
02256
033591
03312
0.267
03527
0.393
0.2881
0.3432
04249
BL3042
04227
0.3524
0.2809
03558
0.2510
0.440
1.125
08733
01161
01445
0 6.1977
011161
0.3876
0.1825
0.2441
6.1114
01911
02211
012115
Irppattton
0.375
0.5
03412
03155
02665
03854
0.3417
0.3758
0.331
0.3655
0.5100
0.359
0.4144
Avg
R1133
0.1914
0.799
0.2155
0,2555
02553
0.3967
03312
0.2588
0.3393
0.393
0.2537
+0.149
Eletriatt Thatl
Roz
0,5911
0347
04444
0
0.500
0 0.665
+0.5281
0.6643
0.4455
0,4914
Delectiee
FUL
0.486
03003
03084
02114
02798
2227
103425
0.332
0.3015
0.265
Pie.
6.450)
03874
03347
03105
03394
0
03360
10.3559
03341
6.1719
Clock Ascenly
Rei
6.781
07911
07355
0
D6966
0.026
0.779
+0.7356
0.571
0.5433
0.503
0.485
0/4111
0/4661
0.4842
40.5055
Pic.
0.427
10.3365
0.951
0
07430
0.038
0.597
0.5850
0
0,0005
Appllanze Dleuctio
1115
B.7842
08412
DARIC
0.9555
(15579
015566
025662
P1
08858
0.212
03385
0.5445
033655
0.5382
015900
0.007
0.5254
Fidterly Alvee
83788
0.7577
10.7555
0.656
07564
0.631
0.6796
0.660
B.4061
10.1779
0.9144
0.4109
D44520
03 0.565
0.1192
0.3055
0.3400
0.2931
0.776l
10.549
0.718
0.5171
0.530
0.4367
0.45%6
0.4611
64212
0.3050
Gerder 13.55
03571
0.2122
10.000
0.540
06338
0,5902
014022
0.590
0500
0,4/95
Age CT.
Asc
0661
0.789
0.001
05774
0.514
0.566
0.177
0.5187
Farnly Stracan 13.5
Acc.
65466
0.9014
0.069
06119
0.5015
01062
B15663
01008
0,454
3.3 Main Results
Overview. As foundation model for power systems, PowerPM achieves SOTA performance on various tasks when compared to other baseline models, highlighting its ability to generalize effectively across wide range of tasks. We derive more detailed comparisons of each task in the following paragraphs, and in all tables we mark the best results in bold, the second-bes best in underlined, and the third-best in "asterisk in each column.
Demand-side Management. The forecasting results for load and solar generation are presented in Tab. I (upper part). The results cover various forecast horizons, including 4 (1 hour), 96 (1 day). 288
7

---

(3 days), and 672 (1 week). The choice of these forecast horizons holds physical significance as it aligns with real-world scenarios. The results demonstrate that not only PowerPM achieves near SOTA performance, but also PowerPMfrect freeze surpasses most baseline models. This highlights the superiority of PowerPM in modeling temporal dependencies and capturing the impact of exogenous variables through the use of a temporal encoder and a novel masked ETS modeling approach. Furthermore, PowerPM attains near SOTA performance at different hierarchical levels, particularly at the macro level (district and city), highlighting the importance of modeling the hierarchical correlation within ETS data in PowerPM. Notably, among the baselines, none of the baselines capture the hierarchical correlation of ETS data, resulting in performance decrease in comparison to PowerPM.
Grid Stability. To assess the efficacy of PowerPM in grid stability application, we con- duct comprehensive experiments encompassing load imputation across various masked ratios (12.5%, 25%,37.5%% 37.5%, 50%), anomaly detection (including electricity theft and clock anomaly de- tection), encompassing total of 18 tasks. The results, detailed in Tab. (middle part), illustrate PowerPM's consistent superiority over all baselines, with the PowerPM freeze variant also surpassing the majority of baselines. Notably, in imputation tasks, PowerPM demonstrates marked superiority over other pre-trained models (such as PatchTST and CoST), underscoring the advantages of hierar- chical modeling in ETS data. Furthermore, in anomaly detection tasks, as shown in Tab. (middle part), our model consistently achieves near-optimal results. While GPT4TS records the highest F0.5 score among the baseline methods, attributed to its generation of GPT-2, PowerPM further enhances the FO.5 score over GPT4TS. This improvement stems from our temporal encoder's 's broader receptive field and the hierarchical encoder's 's capacity to capture hierarchical correlations across all levels, which are both pivotal for modeling ETS data.
Consumer Behavior Analysis. We explore two anomaly detection tasks: elderly living alone and high-power appliance detection, and three classification tasks: gender, age, and family structure classification. The results in Tab. (bottom part) demonstrate PowerPM's SOTA performance, illus- trating its capacity for deep semantic insight and contextual awareness. Furthermore, PowerPM freese sustains high performance, highlighting the model's innate ability to extract and generalize features.
Table 2: Performance comparison on 4 public dataset.
Domwer
Tunk
Po
PowerPM
GPT4TS
TaneLLM TY
UniTise
PicchTST
CoST
T52Nec DM
TiceeNN
Dluner Sk
12
8.2468
0.3162
0,3519
0,3631
01318T
10.3167
0.3565
D/4143
0.3694
D.4173
Stre
26
0.3341
303102
0.3857
+0,3706
0.5755
0.364]
04451
0.453
0.4235
0.4887
Freessslax
id8
0.3767
0.3907
0.138
0.4531
0.4754
0.559L
CAISO
Arg.
0.3688
12
8.1877
0.1195
10.223
02318
02252
0.2688
0.2993
0.3099
0.5001
0.7835
Aree
24
0.2173
07225
15576
03551
0.3055
0.3330
5.3300
0.599
Foocostng
0.319%
00.2555
02664
0350
0.3401
12
BU975
101125
614
0.1541
01 1050
B 11123
0.364)
0.195
1857
Stree
26
0.1134
01895
M01410
111
6.
0.3426
0.3665
Foocusting
168
11794
0:3317
NYISO
0.1195
10.145
0,1654
0.1501
12
*0.0052
0.0066
0,1086
6.854
10225
0.1462
0.1663
0.1913
0.1000
0.1085
Asce
24
0.1154
01567
101193
6.1077
0.334
01182
0.1915
Tomozeting
017347
0.1428
0.136
50.3995
0. 8.1751
0.3307
03001
12
0.1994
102328
02230
02382
0.447
0.2821
0.3176
0.359
0.5261
0.2065
Resion
0.3621
041855
169
03300
0,300
ISONE
Arg.
0.3481
0.29%
102016
02967
0.3039
0.3410
0.3146
044023
0.3033
D.4265
12
0.1289
1564
01736
01M0
10 1016
02152
0.3207
5.3751
0.3557
Froecoting
168
M.2201
02215
02672
040053
0.6545
0.1713
402106
0.200
62359
0.2610
0.3701
0.343
0.2514
0.1924
0.2318
6 6.544
0.357
0.554
285
0,0194
0,312
0440
0.4555
PMM
Avg
0,3389
03557
03333
003362
0.347
0.4065
0,3923
D4363
0.4457
0.4645
12
10,3130
0.095
6.2745
0/127
04107
044003
Timoceting
256
0.4599
6.1661
01 014517
B-4641
044822
0.113
019031
015341
A0g
(1.749
9033651
0.8835
6330
01 19000
0.4170
0.4479
046624
014645
014927
3.4 Model Analysis
Generalization Ability Analysis. To further verify the generalization ability of PowerPM on more datasets from other domains, we evaluate PowerPM on public datasets mentioned above. The results in Tab. 2 demonstrate that PowerPM outperforms nearly all SOTA methods and PowerPMfreest surpasses most SOTA methods, highlighting the generalization superiority of PowerPM.
8

---

Ablation Study, To assess the ef- fectiveness of each component in our model, we conduct several ablation experiments. Specifically, we remove the following components from our model to examine their effects on per- formance: the hierarchical encoder (PowerPM-HD, the dual-view con- trastive learning strategy (PowerPM- C). and the exogenous variables en- coding module (PowerPM-E) Fur- thermore, we replace the masked ETS modeling module with vanilla ran- dom masking (PowerPM-MV), We cat- egorize 44 tasks into four traditional time series tasks: forecasting, missing value imputation, anomaly detection, and classification. The evaluation met- rics are Mean Squared Error (MSE) for forecasting and missing value im- putation, FO.5 score is for anomaly detection, and accuracy (Acc.) for classification. The performance is av- eraged to provide a comprehensive as- sessment.
Ablation Study
Few-shot Learning
Model Scale
Forecasting
1:36
039
MSE
8.55
Parch
MSE
0.59
MSF
Tindk DM
0.45
8.32
0.25
0.39
Imputation
0332
0.244
9.23
MSE
6.475
0.424
MAE
MSE
8.24
2222
6.11
0:20
Anomally Delection
9.44
FILS
0.50
Res
0.50
6461
0.39
0.25
9.42
0.65
Clavsiication
8.70
Ave.
8.6
1.60
6411
855
0:58
0.25
Data Proportion (%)
10
Model Sizee (M) (c)
12)
290
(a)
(b)
Figure 4: Model Analysis: Ablation Study, Few-shot Learn- ing, and Model Scale Evaluation
The results of the ablation study are in Fig. 4 (a). The results indicate that PowerPM outperforms its vari- ants, providing evidence for the contribution of each component. Among them, PowerPM-H exhibits the most substantial decrease in performance compared to the full PowerPM, emphasizing the sig- nificance of interactions between micro- and macro-levels when modeling hierarchical ETS data. The observed performance degradation of PowerPM-M, particularly in forecasting tasks, shows that causal masking can capture more complex temporal dependency. Moreover, the declined performance of PowerPM-C, particularly in anomaly detection and classification tasks, suggests that dual-view contrastive learning is effective in capturing subtle discrepancies between instances. Furthermore, PowerPM-E also presents performance degradation. This emphasizes the effectiveness of the exoge- nous variables encoding module in capturing the impact of exogenous factors. For detailed results of 44 tasks, please refer to App.
Few-shot Learning. In power systems, collecting abundant ETS data for downstream tasks is a significant investment. To demonstrate the practical application value of our work, we conduct a performance comparison between PowerPM and baseline models, considering the limited availability of ETS data. Specifically, models are fine-tuned on 10%, 30% and 60% of the downstream dataset, respectively. Similar to an ablation study, we group our results by task type, which can be seen in Fig. 4 (b). The performance of PowerPM exhibits slight decrease when there is a significant reduction in the proportion of fine-tuning data. This observation serves as evidence of the effectiveness of our novel pre-training strategy. Additionally, it highlights that the PowerPM adeptly captures temporal dependencies and hierarchical correlations present in the ETS data during pre-trainin training, enabling easier adaptation to downstream tasks. More detailed results can be referred in App. 8
Model Scale Evaluation. To explore the impact of model size on performance, we design three variants of PowerPM (about 250M) with smaller sizes: PowerPM- Tiny (about 30M) PowerPM- Small (about 70M), PooerPM-Mediiee
(about 120M), and pre-train them on the same datasets. For the pre- treiirain details, please refer to App. B.I After pre-training, we evaluate these variants on all downstream tasks and present the results by task type like the ablation study, As shown in Fig. 4 (c), as the size of the model increases, we observe overall improvement of the performance on all downstream tasks. Specifically, PowerPM outperforms the other variants all metrics. In addition, larger models exhibit almost decrease in standard deviation, indicating more stable performance.

---

Therefore, the utilization of a larger model with higher capacity and large ETS data enables better generalization across a wide range of downstream tasks.
Related Work
Self-supervised Pre-training Model. Large-scale model based self-supervised pre-training has become more sigmificant in both industrial and academic domains due to the versatility and impressive performance. It initially developed in the fields of computer vision [14] and natural language processing [8] TT.. Self-supervised pre-training in time series is typically classified into
two paradigms: contrastive learning and mask modeling. The objective of contrastive learning is to learn representation by pushing positive pairs closer and pull negative pairs away in the embedding space [16]. Ve2 [2] proposes contextual consistency for positive pair selection. Then, CoST (37] extracts the trend and seasonal feature representations, and takes advantage of both time and frequency domain contrastive loss encourage discriminative seasonal representation. And TF-C 44 applies fime-frequency consistency for embedding time-based and frequency-basee bbsed neighbors. In mask modeling, extract the contextual semantic information, PatchTST 21 masks at the series-level.
Supervised Learning Model. Since the self-attention mechanism in Transformer [33 showed the great ability seize global dependencies between input and output, recently many variants have been proposed to tackle power system tasks. LogTraas 19]. Informer
[88] reduce the complexity by opti- mizing the vanilla self- self-attention mechanism. Autoformer 339 leverages auto-correlation mechanism to achieve series-wise representation aggregation. FEDformer (50 incorporates rreueencydomall
information enhances prediction performance while reducing complexity to linear levels. Besides, DLinear [4 questions the effectiveness of transformers as it outperforms most Transformer-bent bbased SOTAs, with simple linear model. TimesNet 38 has treated time series 2D signal and utilized a convolution based inception net backbone to function as a comprehensive time series model.
Large Language Models Enhanced Model. Recently, the advancement of Large Language Models (LLMs) has opened up new horizons in time series modeling. Many LLMs, such as Ilama [32], GPT- 3 [1I), GPT-4 [1. ChatGLM [9] have the capability to capture complex dependencies and understand varied textual data, yielding sensible reasonable generation results. Therefore, many reserachers begin to apply LLMs to assist time series modeling. Time-LLM [17] and TEXT 28 employ reprogrammed input time series with text prototype embedding and incorporate textual prompts for time series. GPT4TS (51 and UniTime [20] apply fine-tuning to selected components of LLMS improve performance in time series analysis tasks. TEMPO K incorporates the decomposition of time series and betree prompt design for non-stationary time series data.
However, despite numerous methods for sulf-supervised and supervised time series, the research on foundation models specifically designed for power systems remains relatively sparse. And LLMs are limited in power systems scenario, lacking enough textual descriptions for domain knowledge.
Conclusion
This paper introduces the PowerPM, a foundational model designed model ETS data within power systems. PowerPM consists of a temporal encoder and hierarchical encoder. Furthermore, Pow- erPM leverages a novel self-supervised pre-training framework consisting of masked ETS modeling and dual-view contrastive learning. Our experiments involve two real-world scenario datasets. com- prising private and public data. Through pre-training on massive ETS data, PowerPM achieves SOTA performance on diverse downstream tasks within the private dataset. Moreover, when transferred to the public dataset, PowerPM maintains its superiority, showcasing its remarkable generalization ability across various tasks and domains. Further analysis shows the effectiveness of foundation model in the field of power system. Also, PowerPM is an off-the-shelf model with its code and weights. This feature greatly mitigates the challenges associated with sample and label efficiency, allowing it to be directly integrated into various power system applications.
10

---

Acknowledgments
This work was partially supported by National Natural Science Foundation of China (No. 62322606.
No. 62441605).
References
[1] Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, 1lge Akkaya, Florencia Leoni Aleman, Diogo Almeida, Janko Altenschmidr, Sam Altman, Shyamal Anadkat, et al. Gpt-4 technical report. arXiv preprint arXiv:2303.0877 087744 2023.
[2] Vadim Arzamasov, Klemens Böhm, and Patrick Jochem. Towards concise models of grid stability, In 2018 IEEE international conference on communications, control, and computing technologies for smart grids (Smar(GridComm)) pages 1-6. IEEE, 2018.
[3] Defu Cao, Furong Jia, Sercan O Arik, Tomas Pfister, Yixiang Zheng, Wen Ye, and Yan Liu. Tempo: Prompt-hased generative pre-trained transformer for time series forecasting. arXiv preprint arXiv:23 044444 2023.
[4] Widyaning Chandramitasari. Bobby Kurniawan, and Shigeru Fujimura. Building deep neural network model for short term electricity consumption forecasting. In 2018 International Symposium on Advanced Intelligent Informatics (SAIN), pages 43 48. IEFE, 2018.
[5] Ting Chen, Simon Korbllihh, Mohammad Norouzi, and Geoffrey Hinton. A simple framework for contrastive learning of visual representations. In ICML, 2020.
[6] Xinlei Chen and Kaiming He. Exploring simple siamese representation learning. In Proceedings of the IEEE/CVF conference on computer vision and paltern recognition, pages 15750-15758, 2021.
[7] Yue Cui, Kai Zheng, Dingshan Cui, Jiandong Xie, Liwei Deng, Feiteng Huang, and Xiaofang Zhou. Metro: A generic graph neural network framework for multivariate time series forecasting. Proc. VLDB Endow., 2021.
[8] Jacob Devlin, Ming-Wes Wei Chang, Kenton Lee, and Kristina Toutanova. BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding. NAACL, 2018.
[9] Zhengxiao Du, Yujie Qian, Xiao Liu, Ming Ding, Jiezhong Qiu, Zhilin Yang, and Jie Tang. Glm: General language model pretraining with autoregressive blank infilling, arXiv preprint arXiv:2103.10360 2021.
[10] Xi Fang, Satyajayant Misra, Guoliang Xue, and Dejun Yang. Smart grid-the the new and improved power grid: A survey. IEEE communications surveys & tutorials, 14(4):944-98) 201 1.
[11] Tianyu Gao, Adam Fisch, and Danqi Chen. Making pre-trained language models better few-shot learners. IJCNLP, 2020.
[12] Benjamin Goehry, Yannig Goude, Pascal Massart, and Jean-Michel Poggi. Aggregation of multi-scale experts for bottom-up load forecasting. IEEE Transactions on Smarl Grid, 2020.
[13] Mohammad Hashemi, Shengbo Gong, Juntong Ni, Wenqi Fan, B. Aditya Prakash, and Wei Jin. A comprehensive survey on graph reduction: Sparsification, coarsening, and condensation, 2024.
[14] Kaiming He, Xinlei Chen, Saining Xie, Yanghao Li, Piotr Dollár, and Ross Girshick. Masked autoencoders are scalable vision learners. CVPR, 2022.
[15] Wenjic Hu, Yang Yang, Jianbo Wang, Xuanwen Huang, and Ziqiang Cheng. Understanding clectricity-theft behavior via multi-source data. In Proceedings of The Web Conference 2020, pages 2264-2274, 2020.
[16] Ashish Jaiswal, Ashwin Ramesh Babu, Mohammad Zaki Zadeh, Debapriya Banerjee, and Fillia Makedon. A survey contrastive self-supervised learning. Technologies, 9(1):2, 2020.

---

[17] Ming Jin, Shiyu Wang, Lintao Ma. Zhixuan Chu, James Y Zhang, Xiaoming Shi. Pin- Yu Chen, Yuxuan Liang, Yunn-Fang Li, Shirui Pan, et al. Time-Ilm: Time series forecasting by reprogramming large language models. arXiv preprint arXiv:2310.01728, 2023.
[18] Diederik P. Kingma and Jimmy Ba. Adam: A method for stochastic optimization. In ICLR (Poster), 2015.
[19] Shiyang Li, Xiaoyong Jin, Yao Xuan, Xiyou Zhou, Wenhu Chen, Yu-Xiang Wang, and Xifeng Yan. Enhancing the locality and breaking the memory bottleneck of transformer on time series forecasting. Advances in neural information processing systems, 32, 2019.
[20] Xu Liu, Junfeng Hu, Yuan Li, Shizhe Diao, Yuxuan Liang, Bryan Hooi, and Roger Zimmermann. Unitime: A language-empowered unified model for cross doms-n time series forecasting. In Proceedings of the ACM Web Conference 2024, 2024.
[21] Yuqi Nie, Nam H Nguyen, Phanwadee Sinthong, and Jayant Kalagnanam. A time series is worth 64 words: Long-term forecasting with transformers. ICLR, 2023.
[22] Peter Palensky and Dietmar Dietrich. Demand side management: Demand response, intelligent energy systems, and smart loads. IEEE transactions on industrial informatics, 7(3):381-388) 2011.
[23] Yue Pang, Bo Yao, Xiangdong Zhou, Yong Zhang, Yiming Xu, and Zijing Tan. Hierarchical electricity time series forecasting for integrating consumption patterns analysis and aggregation consistency. In IJCAI, pages 1506-3512,, 2018.
[24] Adam Paszke, S. Gross, Francisco Massa, A. Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Z. Lin, N. Gimelshein, L. Antiga, Alban Desmaison, Andreas Köpf, Edward Yang. Zach De Vito, Martin Raison, Alykhan Tejani, Sasank Chilamkurthy, Benoit Steiner, Lu Fang. Junjie Bai, and Soumith Chintala. Pytorch: An imperative style, tigh-performnnc deep learning library. In NewrlPS, 2019.
[25] Michael Schlichtkrull, Thomas N Kipf, Peter Bloem, Rianne Van Den Berg, Ivan Titov, and Max Welling. Modeling relational data with graph convolutional networks. The Semantic Web: 15th International Conference, ESWC 2018, Heraklion, Crete, Greece, June 3-7, 2018, Proceedings 15, pages 593-607. Springer, 2018
[26] Chao Shang, Jie Chen, and Jinbo Bi. Discrete graph structure learning for forecasting multiple time series. International Conference on Learning Representations, 2021.
[27] Arunesh Kumar Singh, S Khatoon, Md Muazzam, DK Chaturvedi, et al. Load forecasting techniques and methodologies: A review. In 2012 2nd International Conference on Power, Control and Embedded Systems, pages 1-10. IEEE, 2012.
[28] Chenxi Sun, Yaliang Li, Hongyan Li, and Shenda Hong. Test: Text prototype aligned embedding to activate Ilm's 's ability for time series. arXiv preprint arXiv:2308..822 2023.
[29] Xiaorong Sun, Peter B. Luh, Kwok W. Cheung, Wei Guan, Laurent D. Michel, S. S. Venkata, and Melanie T. Miller. An efficient approach to short-term load forecasting at the distribution level. IEEE Trunsactions on Power Systems, 2016.
[30] Yuechuan Tao, Jing Qiu, Shuying Lai, Junhua Zhao, and Yusheng Xue. Carbon-oriented elec- tricity network planning and transformation. IEEE Transactions on Power Systems, 36(2):1034- 1048, 2020.
[31] Sana Tonekaboni, Danny Eytan, and Anna Goldenberg. Unsupervised representation learning for time series with temporal neighborhood coding. In International Conference on Learning Representations, 2021.
[32] Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timo- thée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, et Llama: Open and efficient foundation language models. arXiv preprint arXiv:2302.13977. 2023.
12

---

[33] Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention all you need. Advances in neural information processing systems, 30, 2017.
[34] Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, ukasz Kaiser, and Illia Polosukhin. Attention is All you Need. In Advances in Neural Information Processing Systems, 2017.
[35] Hong Wang, Khalid A. Alattas, Ardashir Mohammazzadeh, Mohammad Hosein Sabzalian, Ayman A. Aly, and Amir Mosavi. Comprehensive review of load forecasting with emphasis intelligent computing approaches. Energy Reports, 8, 2022.
[36] Yi Wang. Qixin Chen, Mingyang Sun, Chongging Kang, and Qing Xia. An ensemble forecasting method for the aggregated load with subprofiles. IEEE Transactions Smart Grid, 2018.
[37] Gerald Woo, Chenghao Liu, Doyen Sahoo, Akshat Kumar, and Steven Hoi. Cost: Contrastive learning of disentangled seasonal-trent representations for time series forecasting. ICLR, 2022.
[38] Haixu Wu, Tengge Hu, Yong Liu, Hang Zhou, Jianmin Wang, and Mingsheng Long. Timesnet: Temporal 2d-variation modeling for general time series analysis. In The Eleventh International Conference on Learning Representations, 2022.
[39] Haixu Wu, Jiehui Xu, Jianmin Wang, and Mingsheng Long. Autoformer: Decomposition transformers with cottecoorelltion for long-term series forecasting. NeurlPS, 2021.
[40] Zonghan Wu, Shirui Pan, Guodong Long, Jing Jiang, Xiaojun Chang, and Chengqi Zhang. Connecting the dots: Multivariate time series forecasting with graph neural networks. In Proceedings of the 26th ACM SIGKDD international conference on knowledge discovery & data mining, pages 753-763, 2020.
[41] Dazhi Yang. Gary SW Goh, Siwei Jiang, Allan N Zhang, and Orkan Akcan. Forecast upc-level fmeg demand, part ii: Hierarchical reconciliation. In 2015 ieee international conference on big data (big data), pages 2113-2121 IEEE, 2015.
[42] Zhihan Yue, Yujing Wang, Juanyong Duan, Tianmeng Yang, Congrui Huang, Yunhai Tong, and Bixiong Xu. TS2Vec: Towards Universal Representation of Time Scries. AAAI, 2022.
[43] Ailing Zeng, Muxi Chen, Lei Zhang, and Qiang Xu. Are transformers effective for time series forecasting? In Proceedings of the AAAI conference on artificial intelligence, volume 37, pages 11121-11128 2023.
[44] Chi Zhang and Ran Li. A novel closed-loop clustering algorithm for hierarchical load forecasting. IEEE Transactions on Smart Grid, 2021.
[45] Hao Zhang, Fan Zhang, Yu Zhang. Hui Cheng, Ruotian Gao, Zongpeng Li, Jiakui Zhao, and Mingzhu Zhang. An elderly living-alone guardianship model based on wavelet transform. In 2022 4th International Conference on Power and Energy Technology (ICPET), pages 1249-1253. IEEF, 2022.
[46] Huaying Zhang, Qing Wang, Yan Li, Jingwen Ai, Xunyong Hu, Wenhai Zhang, and Dehai Zhang. Clock anomaly detection method of power quality monitoring device based on voltage sag. In 2021 IEEE 2nd China International Youth Conference on Electrical Engineering (CIYCEE), pages 1-6. IEEE, 2021.
[47] Xiang Zhang, Ziyuan Zhao, Theodoros Tsiligkaridis, and Marinka Zitnik. Self-supervised contrastive pre-training for time series via time-frequency consistency. NeurlPS, 2022.
[48] Haoyi Zhou, Shanghang Zhang, Jieqi Peng. Shuai Zhang, Jianxin Li, Hui Xiong, and Wancai Zhang. Informer: Beyond efficient transformer for long sequence time-series forecasting. In Proceedings of the AÅAI conference on artificial intelligence, volume 35, pages 11106-11115, 2021.
[49] Kaile Zhou and Shanlin Yang. Understanding household energy consumption behavior: The contribution of energy big data analytics. Renewable and Sustainable Energy Reviews, 56:810-
819, 2016.
13

---

[50] Tian Zhou, Ziqing Ma, Qingsong Wen, Xue Wang. Liang Sun, and Rong Jin. Fedformer: Frequency enhanced decomposed transformer for long-term series forecasting. In ICML, 2022.
[51] Tian Zhou, Peisong Niu, Liang Sun, Rong Jin, et al. One fits all: Power general time series analysis by pretrained Im. Advances in neural information processing systems, 36, 2024.
14

---

Table 3: Private dataset description
Dataset
Instance
Samples
Output Length
Frequency
Classes
#city #district
Pre-training
90
268373267040
15 minutes
#user
1530826
#city
11
Load forecasting
#district
90
109596429408
[4, 96, 288, 672) 15 minutes
#user
1563730
#city
11
Load
#district
90
109596429408
672
15 minutes
imputation
#user
1563730
#city
Solar generation forecasting
#district
3458400
[4. 96, 288, 672)
minutes
#user
192
#city
Electricity theft detection
#district
90
279478936
1
Iday
#user
44077
#city
11
Clock error detection
#district
90
1070142528
1
15 minutes
#user
26083
#city #district
11
Elderly alone detection
90
25762488
Iday
#user
35145
#city #district
High-power appliance detection
33402144
day
#user
24972
#city
11
Consumer
#district
90
18661860
Iday
[2, (2,4,4)
analysis
#user
29476
A Dataset Description
We conduct experiments on 5 real-world hierarchical electricity time series datasets, one of which was collected from the real scenario. The other four are collected from CSISO ISONF NYISO and PJM [ Our experiments include four typical time series analysis tasks on these datasets to evaluate the effect of our approach both in-domain and cross-domain settings: prediction, missing value imputation, anomaly detection, and classification, which include diffferent sampling frequencies (5 minutes, 15 minutes, hour, 1 day). Moreover, covers variety of application scenarios in power systems (load forecasting, solar generation forecasting, electricity theft detection and consumer analysis, etc.). Tab, B and Tab. summarizz the detailed descriptions of these datasets.
g
A.1 Private Dataset
Private dataset is collected from the load data in the real scenario, covering the period about 6 years. Following data preprocessing, we extract subset the data. In order to effectively support research objectives, we divide the dataset into distinct sub-datasets. One biggest of these sub-datasets is served as the pre-traiming datasct, while the remaining 7 sub-datasets are utilized as downstream datasets for downstream tasks. These downstream datasets are partitioned into train, validation, and test sets according to 6 2 ratio, ensuring that the training set contain data from the earlier time period. Further details are provided below:
http:///ww...ooeeeccc.e/ee/eseer and and-demand/
http://wwwwiiiiooommoaaaaaaeeeee https://datamiier2.. ppt.ppmmmc.c.....mmmmmm
15

---

Table 4: Public dataset description
Dataset
Instance
Samples
Output Length
Frequency
Time Span
#state
1
CAISO
305018
(12, 24, 168)
hour
2023-04422--2222444--33
#area
34
#region
ISONE
25904
(12, 24, 168)
hour
2023-10-011-2024444-00
#state
6
#state
NYISO
1396992
(12, 24, 168)
5 minutes
2023--03-01122224403--3
#area
11
#state
3
PJM
212369
{12, 144, 288)
5 minutes
2024-03--88-2224--4-22
#ciit
22
Pre-training Dataset. The pre-training dataset is derived from subset of the private dataset, encompassing the period about 4 years.. It consists of unlabeled data recorded at frequency of one data point every 15 minutes. The dataset is structured hierarchically, including information the user, district, and city levels.
Load Forecasting and Missing Value Imputation Dataset. This dataset is extracted from portion of the private dataset about years. The dataset includes hierarchical information at the user, district, and city levels, with data points recorded every 15 minutes. For the missing value imputation task, the dataset is structured to output 672 data points. As for the forecasting task, there are four different hour (4 data points), one day (96 data points), three days (288 data points), and seven days (672 data points).
prediction horizons:
Solar Generation Forecasting Dataset. The dataset is collected from many distributed photovoltaic power stations. The dataset has not hicrarchical structure, and data points are recorded at frequency of one point every 15 minutes. It includes four different prediction horizons: one hour, one day, three days, and seven days.
Electricity Theft Detection Dataset. This dataset comprises the daily electricity consumption records (in K- Wh) users in 1 year. For each user, the dataset includes the daily aggregate electricity usage. Within the dataset, certain users (referred to as electricity thieves) engage in unauthorized activities involving the electricity meter order reduce costs.
Clock Anomaly Dataset. This dataset comprises millions of clock error series, each representing the time deviation, compared the standard time, and communication delay of various watt-hour meters on weekly basis. The dataset covers the period about 8 months.
Elderly Living Alone Dataset. This dataset includes the daily electricity consumption records (in K-Wh) of village users. Additionally, employees conduct extensive on-site investigations specifically targeting these users, from which obtain labels indicating whether each user is an elderly individual living alone or not.
High-power Appliance Detection Dataset. This dataset consists of the daily electricity consumption records (in K. K-Wh) of village users. Similar to the previous dataset, on-site investigations are conducted by same method, enabling to collect labels indicating whether each user possesses high-power appliances.
Consumer Analysis Dataset. This dataset contains the daily electricity consumption records (in K-Wh) of village users. Additionally, employees conducted extensive on-site investigations targeting these users, collecting statistics related to the gender of the gender of user who lives alone, the age of the resident elderly, and family structure. The gender labels of user who lives alone are: male and female, totaling two classes; the age labels for residents are: 60 ~ 70 years old, 70 80 years old, 80 90 years old, and over 90 years old, totaling four classes; the family structure labels are: people, ~ 3 people, 4 ~ 5 people, and more than 6 people, totaling four classes.
16

---

A.2 Public Datasets
Four public datasets as cross-domain datasets are selected to validate the generalization ability of our model. These four datasets are named CSISO, ISONE, NYISO, and PJM, which cover 3 types different hierarchical relationships: state-area, region-state, state-city.
CAISO. It is sampled from California, including 34 areas loads and an aggregated load for the state, recorded every hour from April 25, 2023, to April 23, 2024. The prediction horizons include half day (12 points), one day (24 points), and seven days (168 points).
ISONE. It is sampled from New England, consisting of 6 states loads and an aggregated load for the region, recorded every hour from October 1, 2023, to April 1, 2024. The prediction horizons include half a day (12 points), one day (24 points), and seven days (168 points).
NYISO.
is sampled from California, containing 11 areas loads and an aggregated load for the state, recorded every 5 minutes from March 1, 2023, to March 31, 2024. The prediction horizons include one hour (12 points), half a day (144 points), and one day (288 points).
PJM. It is sampled from 3 states: Florida, Ohio, Washington, which includes 22 cities loads and there 3 state loads, recorded every hour from March 28, 2023, to April 26, 2024. The prediction horizons include one hour (12 points), half a day (144 points), and one day (288 points).
A.3 Exogenous Variables
We obtained weather and temperature records for all area levels in both the private and public datasets. The weather information from the private dataset is obtained from the Weather Rada TT) Additionally, the weather information from the public datasets is obtained from the NSF NCAR Research Data " !1 Both sources cover the same timespan as mentioned above, respectively. These records include the maximum and minimum temperatures (in "C for private dataset and 'F for public datasets)
for each hour in each city.
B PowerPM and Baseline Implementation Details
B.1 PowerPM Implementation
The pre-training stage of the experiment is implemented in PyTorch (24] and conducted on a Linux system with 2 CPUs (AMD EPYC 9654 96-Core Processor) and 8 GPUs (NVIDIA Tesla A800 80G) for about 8 days. And the downstream task experiment is repeated five times. We select 512 samples as a batch, and every batch contains about 174k patches, which set patch len to 48, stride to 24. To speed up the model training, we stop the gradient update of the background nodes in the hierarchical graph. We optimize with Adam [18, updating the model parameters every 4 steps, and the model trains for 1310k updates in total. A reduce learning rate on plateau scheduler is utilized to adjust learning rate during pre-training. Specifically, we set the basic learning rate le and the maximum learning rate as 2e - 5. and the learning rate updates for every 10k updates. In addition, we trained three additional variants of PowerPM with different parameter counts to meet the needs of different users or situations. Detailed model hyperparameters can be found in Tab. 5
Full Fine-tuning. In the F-FT (Full Fine-tuming) setup, for different tasks, introduce different head И on the top of pre-trained encoder f(). where both the parameters of the encoder f() and the head H are trainable. For forecasting and imputation tasks, we use a prediction H; head to map prediction points or reconstruction points from 2, In this setup, we fine-tune both the head and the encoder f()). We utilize 100%, 60%, 30% and 10% training data for fine-tuning. a one-layer fully connected network to implement prediction Hi and logistic regression from the Sklearn library to implement the classifier H.. The learning rates are specifically set to 4e 4 and
utilize
3e - 5 for public and private datasets.
Partial Fine-tuning. In the P-FT (Partial Fine-tuning) setup, for different tasks, we also introduce different head И on the top of pre-trained encoder f(.). For forecasting and imputation tasks, we use prediction H, head to map prediction points or reconstruction points from 2,, And for anomaly
10 weather com. cn/ https: ucar edu/
17

---

detection and classfication tasks, a classifier He on top of the pre-trained encoder f(.). During the whole finetune process, we keep the parameters of f() fixed. Only the head is fine-tuned in this setup. we utilize a one-layer fully connected network to implement prediction H₂ and logistic regression from the Sklearn library implement the classifier H The learning rates are specifically set to 4e 4 and 3e 5 for public and private datasets.
B.2 Baselines Implementation
We compare with 8 state-of-the-art methods: including Large Language Model (LLM) enhanced models: GPT4TS [ST] Time-LLM (77], UniTime (20]: pre-train models: PatchTST (2], CoST 37. Ve2 [2]; supervised models: DLinear [43), TimesNet (38). To make fair and comprehensive comparison, we reproduce all models with official implementation, and use different output head for different downstream tasks. Due to the large scale of the ETS dataset, we increase the number of training epoch and reduce the learning rate in order to make the parameters of the model fully learned.
GPT4TS [51 combines the LLM with Transformer, which use frozen pre-trained GPT-2 for general time series analysis. To implement GPT4TS, we utilized their open-source code, available at wttpa//////...--...-ee.e.... -in- We use the 6 layers of GPT-2, which is proved to have the optimal performance in original paper and the total size of GPT4TS is about 105.15M, and the trainable parameters are 24.04M (GPT-2 is frozen). We set the number of train epochs to 50, the learning rate to 0.0005, and the batch size to 256.
Time-LLM 17 frezees the LLM the backbone, and align time series to text with patch reprogram- ming. It also designs Prompt-as-Prefit including dataset context, task instruction and input statistics to enrich the input context direct the transformation of reprogrammed input. We utilized their open-source code, available at htttp:://githubb.c.
KimMeen/Tingg LLM to implement Time-LI L. We set the 7lama-7b with 32 layers as the backbone, which is the most effective recorded in [17] and the total size of Time-LLM is about B, and the trainable parameters are 58.55M (llama-7b is frozen). To align the dataset context input to our datasets, we constuct different natural language prompt summarized in App. A for private and public datasets, and we set the number of train epochs to 50, the learning rate 0.005, and the batch size 256.
UniTime 20 leverages LLM to handle time series forecasting across time series domains, which exhibit significant differences in temporal patterns and distribution. The same dataset context in Time-LLM, UniTime also designs human-crafted instructions to furnish the model with explicit utilized their open-source code, We implement the backbone LLM with GPT2-small like original paper, and the total size of UniTime is about 108. 54M without freeze any parameters. We use the same natural language prompt in Time-LLM as the human-crafted instructions for different datasets, and set the number of train epochs to 50, the learning rate 0.0005, the weight decay to 0,0001, and the batch size to 256.
domain identification information. To implement UniTime, available at http:://gttubb.co. wwwm/iixu///mmi//
TS2Vec [42
performs contextual consistency using overlapping subseries and hierarchical loss function to capture data consistency at the observation and sample levels. We utilize the open-source code available at http:://tthub.com//hhaaayyyy cow//zhhhaanuuee//eeeeee Specifically, we set the number of epochs for pre-training to 100, the learning rate to 0.0005, and the batch size to 512. Due to the large scale and complex semantics of the pre-trained ETS data, we adjust the representation dimension to 640, matching the ETS data characteristics. We adopt the default settings provided by the TS2Vec implementation for other settings during pre-training.
CoST [37] comprises both time domain and frequency domain contrastive losses to learn dis- criminative trend and seasonal representations. We utilize the open-source code available at https://githu..cm// wow.sallssooorr/CCSTTTTTT to implement CoST. Specifically, we set the number of epochs for pre-training to 100, the learning rate to 0.0005, representation dimension to 640, and the batch size to 256. We adopt the default settings provided by the CoST implementation for other settings during pre-training.
TST 21 changes the input sequence as a series of patch windows, focus the subseries-level attention to capture local semantic information while minimizing memory consumption. We utilize the open-source code available at httpg/gghhhhooogggiiiin/////eeee PST We set the patch len to 32 and stride to 16, the number of epochs for pre- pre-training to 100,
For hyperparameters of
18

---

the learning rate to 0.0005, and the batch size to 512. We adopt the default settings provided by the PatchTST implementation for other settings during pre-training.
TimeNet (38 a CNN based time series model which extends the analysis of temporal variations into the 2D space. It designs TomesBlock with inception block to extract complex temporal patterns, leading to multiple time series tasks. To implement TimesNet, we utilized their open-source code, available https::/github.coom// htt.:/ooo/t/miiiiim -
Specifically, we set the number of epochs for training to 50, the learning rate to 0.0005, and the batch size to 128. We adopt the default settings provided by the TimesNet implementation for other settings for forecasting, imputation classfication anomaly detection.
DLinear
4 decomposes the time series into trend sequence and a seasonal sequence, then model these two sequences using two simple MIPs. To implement Diinear, we utilized their open-source Specifically,
the numher of epochs for training to 50, the learning rate to 0.0005, and the batch size to 512. We adopt the default settings provided by the DLinear implementation for other settings.
code, available at
B.3 Cluster Method
We use K-means algorithm to cluster users. Firstly, we get filter out user ETS by labels, and normalize the time series data, represented as N x M matrix, to ensure that differences in scale do not affect the clustering results. Next, we use DTW as the distance metric to cope with time shifts and different rate variations in ETS data and randomly initialize cluster centers. By calculating the distance from each time series to each cluster center, assigned to the nearest cluster center, and the cluster center is recalculated according the assignment result,and the process is iterated until the cluster center is stable. We experimented 10 times with different K, and used elbow method to select the optimal number of clusters, and finally determined 12.
C
Full Results
Due to the limited length of the text, we summarize all the experiments in the main text into two parts: the main experiment and the analytical experiment. We categorize and index them in Table 6 7 g
D Limitations
PowerPM is designed for electricity time series modeling, containing about 250M parameters. As foundation model, although we have provided relatively comprehensive results to verify the model's effectiveness, the model still exsits limitations. In fact, there various kinds of ETS in the power systems, which contain not only the electricity consumption data generated by human activities, but also the sequence generated by system operation and sensor detection. In this paper, PowerPM only pre-train on load data. In the future, by increasing model parameters and improving model architecture, will use more kinds of ETS data for training, that it capture more complicated ETS semantic information, understand more complex power system operation rules, and provide more complete help for power systems.
E
Social Impacts
This paper presents PowerPM as foundation model for power systems and has been deployed in the real scenario. It focus on demand-side management, grid stability and consumer behavior analysis, providing the possibility to understand and analyze electricity time series. There is no potential ethical risk or negative social impact.
19

---

Table 5: The model hyperparameters of PowerPM with different model size.
Parameter
PowerPM
PowerPM-Medium
PowerPM-Small
Power
Model Scale
256.0M
120.1M
68.6.M
35.5M
Temporal Encoder
26
18
12
4
Model Dimention
1024
768
768
768
Inner Dimension
2048
2048
1024
768
Hicrarchical Encoder Layer
2
2
2
2
Heads
16
16
16
16
Mask Ratio
0.4
0.4
0.4
0,4
Time Shift
96
96
96
96
Number of Clusters K
12
12
12
12
Batch Size
512
256
256
128
Learning Rate
1c-6
lc-6
2c-6
2c-6
Optimizer
Adam
Adam
Adam
Adam
Scheduler
Plateau
Plateau
Plateau
Plateau
Table 6: Additional performance comparison on private dataset in terms of MAE metric. Forecasting tasks involve varying forecasting lengths {4, 96, 288, 672) time points and imputation tasks involve varying mask ratio {0.1 125, 0.25, 0.375, 0.5). The length of the input window is 672.
PowerrM
PrweetMM yrares
CIPEATS
M
Uniflime
PacckNT
CAST
ThmmsNNi
L
MAE
MAE
4
0,3638
0
0.424
0.893
0.4166
B.4255
0,4412
0.4350
0.4512
0.4540
Excluive User
Furocattng
672
288
96
8.4896
0.432
0.4352
0.9012
0.4553
019657
0.3357
115157
010963
0.3154
6:4503
0,2759
115455
01515
0,5512
4
0.3M51
1.009
6 5945
0.4215
0.4525
0.401
0:4300
96
0,35M0
022277
04553
84128
0.436
0.3622
0.3136
0.574
0.315
0.310
Forecastin
285
0.4114
B.4569
0.474
0.3122
044677
0.4455
0.5357
0.007
0.473
0.5495
0,3030
0,3985
04120
0.3928
0,416
DJ5555
0.4255
0.6500
0,2014
04496
96
0.3719
0:519)
05110
014674
0.9349
672
0.4541
0
05250
0.410
Avg.
0.4031
014633
0.4479
0.509
0.5052
0.529)
0.5385
0,479
0,5577
0.1619
0.302
02643
0.5467
02221
03191
Forecasing
612
285
6.2991
0.2471
1 13443
0.5855
0.3364
0.3775
0.400
0.5514
1816
0.9944
0.4100
1905
0.206
0.112
0.553
Ang:
0.1283
1.2855
003085
03392
0.1144
8 0.551
6:300
0-4864
02212
04393
Furscastng
672
286
0.3126
0,3765
0 900
014255
0.3597
0.5164
0,4258
0 1557
0.3827
0.9919
1686
0.446
03385
0.4488
01869
0.5531
0.9328
Aag.
0,3759
0.3175
02913
0.3125
03113
03185
0.454
0.350
0.335
0,4970
Excluive User
Inporation
0.375
0.35
0.3017
0.3828
1.511
0.3944
116617
0.3913
0.4167
04313
G-117
04495
0.397
84611
0.4539
0.3366
0.4287
03055
0.4906
0.4288
0.5320
Avg
0.3012
0.390
1604
04430
04516
0.009
0-4549
0.3966
Public Use
Impoction
0.375
0:35
0.5
0.25%
0.3615
0.1991
0.2519
0.1222
0,940
05236
03151
113107
04475
0.38
03009
0.3157
0.450
0.5451
11417%
05148
016655
0.558
05337
0.4679
0.5147
0.6166
Ava.
0.2255
0,296
0.4417
014613
0.5055
0:125
0,1021
0.17MM
0.360
11300
02256
0.3653
Inporation
Disikt
0.375
0.25
0.
0.1543
0.3311
0.1984
02115
11
0.256
0.2258
0.2224
0,2755
0.2644
0.2833
3475
0.2915
03359
0.355)
0.3566
0.3705
0.5455
0.3947
04111
0.3155
04420
0.5500
0/4388
0,3406
0-8873
Avg
6:125
6.6676
0.1499
0.1531
0.1350
0.140
11090
0.2350
0.251
0.2034
0,2715
Impucatioo
City
0.375
0:35
0.5
63733
61224
0,1877
0.2385
0.3027
072141
0.3124
0,3340
0.3865
03032
11555
0.4350
0.3116
0.3866
113535
0.58%
0.275
0.5048
0.3625
0/100
0.390
03373
Avg
20

---

Table 7: Detailed performance of ablation study. Forecasting tasks involve varying fore- casting lengths of {4, 96, 288, 672) time points, imputation tasks involve varying mask ratio {0.125, 0.25, 0.375, 0.5}. The length of the input window is 672.
PowerPM
PowerPM-H
PowerPM-M
PowerPM-C
PowerPM-E
Tasks
MSE
MAP
MSE
MAE
MSE
MAE
MSE
MAB
MSE
MAE
0.3378
0.3638
0.3505
0.3808
0.3777
0.3859
0.3672
0.3776
*0,3531
*0.3788
Exclusive User
96
0,4183
0.449
0.4389
*0.4642
+0.4343
0.4770
0.4253
0.4546
0.4496
0.4650
Foorecasting
288
0.4770
0,4653
0.5061
*0.4879
0.4957
0.4906
+0.4804
0.4885
0.4853
0.4718
672
0.5476
0.5222
+0.5765
0.5494
0,5772
0.5502
0.5957
0.5362
0.5668
90.5371
Avg.
0.4452
0.4502
+0.4680
0,4706
0,4712
0.4759
0.4694
70,4642
0.4637
0.4632
0.2353
0.2951
0.2428
0.3041
0,2793
*0,3024
0.2519
0.3239
*0,2448
0.2977
Public
96
0.2604
0.3190
0,3126
0.3293
0.3029
0,3473
+0.2973
0.3339
0.2966
003325
Demand vide Management
Forecasting
288
0.3226
0,3875
*0.3455
0,4103
0,3480
*0
0.3460
0.3938
0.3334
0.4096
672
0,3818
0,4241
0.4330
0.4683
+0.4003
0.4595
0,3946
-0.431
0.4031
0.4349
Avg.
0.3000
0.3564
0.3335
0.3780
0.3326
0.3785
+0.5225
*0,3737
0.3195
0.3687
4
0.2382
0.3090
*0 +0.2643
0.3394
0,2739
00,322
0,2418
0.3165
0.2714
0.3232
District
96
0.2926
0,3419
0.3454
0,3913
*0,3371
0.3654
0.3278
003699
0,3385
0.3796
Forecasting
288
0.3300
0,3874
0.3767
0,4338
0,3896
0,4015
0.3417
00,488
*033659
0.4190
672
0,3710
0.4241
0.4105
0.4757
+0,3924
0.4682
0,3809
0.4485
0.4038
+0.4583
Avg.
0.3656
0,3492
0.4100
*0.3893
0.3231
0.3881
*0.3449
0.3950
0.1725
0.1639
*0 20:2
0,1710
0.2340
0.1934
0.2123
10.17
0.1941
0.1812
City
96
0.2272
0.2131
0,2669
0,2570
+0,2462
0.2313
0.2336
0,2403
0,2478
0.2415
Forecasting
288
672
0.3211
0,2471
0.3191
0 0.3646
0,1187
0,3820
0.3114
0,3415
0.3119
+0,3498
00,2950
40 3486
0.2670
0,3426
0.2929
00.2713
0.3563
0.3622
0.3054
Avg.
0.2423
0.2358
0.2889
0.2804
0,2834
*02677
0,2654
0.2632
0.2667
0.2726
0.0993
0.1541
10.1115
0.22
0.1117
0.1691
0.1
*0. 1732
Solor Generation
l'orecasting
288
96
0.1223
0.2337
0.2001
0.2526
*0,2637
10,1603
0022270
0.2859
0.2548
0.1412
0,313
0 0,2097
0.1694
0.2713
0.2310
0.3138
672
0.3076
0.3165
0,3616
0.3332
0.3213
0,3373
+0,3562
0.3686
Avg.
0.1907
0.2309
+0,2243
0022572
0.2073
0,2569
0.2270
0.2717
0.125
0.2459
0.2654
0.2666
0,2999
0,2738
*0.2845
2033633
0,2717
0.2508
0.2865
Exclusive User
0.25
0.2621
0.2849
0.3160
0.3165
0.3055
0.3210
+0
0.3117
0.2957
90.3146
Imputation
0.375
0.5
0.3661
0.3288
0.1528
0.3017
0 0:4426
0,3586
0.3555
0,4095
04144
0.3729
0,3892
0.188
0.4421
10,3594
00,340
0.3359
+0.4209
0.3783
0.3723
30.3434
Avg.
0.3007
0.3012
0,3459
0.3454
703316
0.3533
0.3418
0.3258
0.3364
+0.3292
0.125
0.2348
0.1514
0.2633
0,1762
0.2495
10.1777
+0,2484
0.1819
0,2457
0.1841
Public
Imputation
0.375
0:25
0.3237
0.2776
0.2036
0.2392
0.3197
0.3621
0,2179
0.3003
0.3541
0,2884
0.201
0.2945
0.2793
0.3367
0.2652
0.2171
+0,2847
*0,3471
.022168
10.2716
0.5
0.3919
0.3418
0.4485
0,3866
*0,4201
0.3734
0,3833
0 0,3556
0.4288
*03356
Avg.
0.3070
0.2340
0,3484
0,2703
0,3280
0.2639
0.3156
0,2549
*0.3265
*0,2573
0.125
0,0811
0.1021
0.1268
0.1508
0,1185
0.1496
10.1074
10,1140
0.1058
0.1073
Geid Stabili
District
0.25
0,1284
0,1543
0..124
0.2007
0.1505
0.1843
0.1566
0.1576
0,1629
00.1676
Impatation
0.375
0.5
0.1666
0.2269
0.1904
0,2452
0,2753
021888
0.3085
0.2417
*0,2771
0,2147
0.2905
0.2330
0,2864
0.1878
0.3048
0.2115
0.3028
0022033
0.3155
0.2556
Avg.
0.1730
0.1935
0.2254
*0.1902
0.2144
0.1838
1970
0.1937
0022115
0.125
0.0753
0.0876
0.1222
0.1407
0,1078
0.1208
0.0819
10000
+0.0993
0.1009
City
0.25
0.1114
0,1294
0.1688
0.1832
0.1491
0.1549
0,1210
0.1362
00.147
0.1651
Imputation
0.375
0.1451
0.1735
002108
0.2335
0,2362
*0.2136
086
0,1962
0.2253
0.2140
0.5
0.2412
0,2533
0. 305555
0.2943
+0,2742
.0.2715
0,2689
0,2666
0.2957
0.2844
Avg.
0,1610
0.2018
0.2129
*0.1918
*0.1902
0.1651
0.1919
0.1911
Pre.
0,3793
0.3612
*0.3457
0.3068
0.3141
Electricity Theft Detection
Rec.
0.5911
0.5597
0.5175
+0.5288
0.5204
F0.5
0.4086
0.3888
*0.3703
0.3349
0.3412
F1
0.4621
0.4391
*0.4145
0.3883
0.3918
0.4437
Clock Anomaly
Rec.
0.7881
0.7574
*0.7446
0.7184
0.7358
Detection
FO.5
0.4961
0.4838
0,4850
0.4559
+0.8899
F1
0.5761
0,5596
+0.5380
0.5283
0.5560
Pre.
0.7427
0,7364
*0.7130
0.6915
0.7040
Consumer Behavior Analysia
High Power
Rec.
*0.5619
0.5610
0.5452
0.5648
Appliance Detection
FO.5
0.70442
0.6934
*0.6763
0.6563
0.6709
F1
0.6534
0.6374
*0.6279
0.6097
0. 0.6267
Pre.
0.4540
*0.4n97
0.3737
0.3588
0.4121
Elderly Alone
Roc.
0.7881
*0.7551
0,7654
0.6956
0.7293
Detection
FO.5
0.4961
+0.4509
0.4163
0.3972
0.4514
F1
0.5761
0.5311
0,5022
0.4734
+0.5266
Gender CLS
Acc.
0.7571
*0.7169
0.6946
0.7233
0.6854
Age CLS
Aoc.
0.6830
0.6671
0.6515
0.6470
*0.6502
Family Structure CLS
Acc.
0.6406
0.6265
*0.6191
0.6114
0.5815
21

---

Table 8: Complete results of few-shot learning performance comparison. Models are fine-tuned on {10%, 30% and 60% of the downstream dataset. Forecasting tasks involve varying fore- casting lengths of {4, 96, 288, 672) time points and imputation tasks involve varying mask ratio {0.125, 0.25, 0.375, 0.5]. The length of the input window is 672. We average the result for each task.
Model
Tasks
60%
30%
Decrease
10%
Decrease
Forscasting(MSS)
0.4723
0.5553
17.58%
0.6275
32.87%
Imputation(MSE)
0.4021
0.4884
21.46%
0.5739
42.72%
TS2vee
Anomaly Detection(FU.5)
0.4027
0.3454
14.24%
0.3173
21.20%
Classification(Ac))
0.5234
0.4197
19.82%
0.4335
17.17%
Forecasting(MSSE
0.4711
0,5589
18.64%
0.6349
34.78%
Imputation(MSE)
0.3825
0.4704
22.97%
0.5059
32.26%
CoST
Anomaly Detection(F0.5)
0.4221
0.3785
110.34%
0.3156
25.23%
Classification(Aco.
0.5534
0,4806
13.15%
0.4363
21.15%
Forecasting(MSSE)
0.4456
0.5105
14.56%
0.5716
28.29%
PatchTST
Imputation(MSSE)
0.3623
0.4346
19.95%
0.4592
26.76%
Anomaly Detection(FO.5)
0.3452
0.2657
23.03%
0.2283
33.87%
ClasificattonnAAcc.)
0.4526
0.3341
26.18%
0,2808
37.95%
Forecusting(MSE)
0.3904
*0.4220
8.10%
0.4528
15.98%
Imputation(MSSB
0,3375
0.3722
10.29%
0.3895
15.41%
UniTime
Anomaly Detection(F0.5)
0.4102
0,3640
11.26%
0.3391
17.34%
Classification( Acc.)
0.5439
0.4740
12.85%
0.4551
16.33%
Forecasting(MSE)
0.3713
0.4034
*8.64%
0,4180
12.58%
TimeLLM
Imputation(MISE
0.2815
0.3072
9.13%
0,3104
10,27 %
Anomaly Detcction(F0.5)
0.4024
0.3655
9.16%
*0.3534
12.17%
Classfication(Acc.)
0.5417
0.4958
8.48 %
*0.4637
144399%
Forocasting(MISE)
*0.3838
0.4343
13.15%
*0.4447
15.86%
Imputation(MSFF
*0.3212
*0.3614
12.53%
*0.3846
19.75%
GPT4TS
Anomaly Detection(F0.5)
*0.4196
*0.3718
11.39%
0.3587
14.552%
Clasiffcation(Accc))
*0.5483
*0.4902
10.60%
0.4737
13.61%
Forecasting(MSEE
0,3343
0.3551
6.22%
0,3652
9.25%
Imputation(MSEB)
0.2717
0,2998
10.34%
0.3167
16.57%
PowerPM
Anomaly Detection(F0.5)
0.4822
0.4459
7.53%
0.4166
13.60%
Classificatio((Ac..
0.6594
0.5943
9.88%
0.5735
13.02%
22

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification: In this paper, abstract and introduction accurately reflect the paper's contribu- tions and scope.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or
NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.
is fine to include aspirational goals as motivation as long as is clear that these goals are not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations the work performed by the authors?
Answer: [Yes]
Justification:
The limitations of the work discussed in Appendix D
Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create a separate "Limitations" section their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g.. independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect on the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution is low images are taken in low lighting. Or speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.
While the authors might fear that complete honesty about limitations might used by reviewers as grounds for rejection, worse outcome might be that reviewers discover limitations that aren't 't acknowledged judgment and recognize that individual actions in favor of transparency play impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
the paper. The authors should use their best
3. Theory Assumptions and Proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer: [NA]
23

---

Justification:
The paper does not include theoretical results.
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated referenced in the statement of any theorems.
The proof' can either appear the main paper or the supplemental material, but they appear in the supplemental material, the authors are encouraged to provide short proof sketch to provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should properly referenced.
4. Experimental Result Reproducibility
Question: Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?
Answer: [Yes]
Justification:
The private dataset so one may cannot reproduce the experiments on the private dataset. However, we also use another two public datasets, and the code of our work fully provided too, which could help you understand our work.
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is novel architecture, describing the architecture fully might suffice, or if the contribution a specific model and empirical evaluation, it may be necessary either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g., in the case of large language model), releasing of a model checkpoint, or other means that appropriate to the research performed.
While NeurlPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) the contribution is primarily a new algorithm, the paper should make clear how reproduce that algorithm. (b) If the contribution is primarily new model architecture, the paper should describe the architecture clearly and fully.
(c) the contribution is a new model (e.g., a large language model), then there should either be way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should possible for other researchers to have some path to reproducing or verifying the results.
5. Open access to data and code
24

---

Question: Does the paper provide open access to the data and code, with sufficient instruc- tions faithfully reproduce the main experimental results, as described in supplemental material?
Answer: [Yes]
The private dataset so one may cannot reproduce the experiments on the private dataset. However, we also use another four public datasets, and the code of our work is fully provided too, which could help you understand our work.
Justification:
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurlPS code and data submission guidelines (https://nips.co
guuides/CodeSumiisse
for more details.
While we encourage the release of code and data, we understand that this might not be possible, so "No" an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines (https
/
ssiociolicyy for more details.
The authors should provide instructions on data access and preparation, including how access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing much information possible in supplemental material (appended to the paper) is recommended, but including URLs to data and code permitted.
6. Experimental Setting/Details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification: We give detailed description of our experiment setting in Appendix B.
Guidelines:
The answer NA means that the paper does include experiments.
The experimental setting should be presented in the core the paper to level of detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, as supplemental material.
7. Experiment Statistical Significance
Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Question:
Answer:
[Yes]
Justification: The overall standard derivation is shown in the figure
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by error bars, confi- dence intervals, or statistical significance tests, least for the experiments that support the main claims of the paper.
The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall with given experimental conditions).
25

---

The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g., Normally distributed errors).
It should be clear whether the error bar is the standard deviation the standard error of the mean.
It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful figures symmetric error bars that would yield results that are out of range (e.g. negative rates).
show in tables or
If error bars are reported in tables or plots, The authors should explain the text how they were calculated and reference the corresponding figures tables in the text.
8. Experiments Compute Resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?
Question:
Answer: [Yes]
Justification: We provide information about Experiments Compute Resources such as CPU, GPU, etc. in the Appendix B
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make into the paper).
9. Code Of Ethics
Question: Does the research conducted the paper conform, in every respect, with the NeurlPS Code of Ethics
Answer: [Yes]
Justification:
All authors reviewed and conducted the NeurIPS Code of Ethics.
Guidelines:
The answer NA means that the authors have reviewed the NeurIPS Code of Ethics.
If the authors answer No, they should explain the special circumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g., there a special consid- eration due to laws or regulations in their jurisdiction).
10. Broader Impacts
Question: Does the paper discuss both potential positive socictal impacts and negative societal impacts of the work performed?
Answer: [Yes]
Justification: Our potential positive societal impacts is discussed in Appendix E
Guidelines:
The answer NA means that there is no societal impact of the work performed.
If the authors answer NA or No, they should explain why their work has no societal impact why the paper does not address societal impact.
26

---

Examples of negative societal impacts include potential malicious unintended uses (e.g., disinformation, generating fake profiles, surveillance), faireess considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional unintentional) misuse of the technology.
If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms monitor how system learns from feedback over time, improving the efficiency and accessibility of ML).
11. Safeguards
Does the paper deseribe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [No]
Justification:
This paper poses no such risks.
Guidelines:
The answer NA means that the paper poses no such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?
Question:
Answer: [Yes]
Justification: All the existing assets are properly referenced.
Guidelines:
The answer NA means that the paper does use existing assets.
The authors should cite the original paper that produced the code package dataset.
The authors should state which version of the asset used and, possible, include URL.
The name of the license (e.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (e.g., website). the copyright and terms of service of that source should be provided.
If assets are released, the license, copyright information, and terms use in package should be provided. For popular datasets, paperswithcone com/ datasets has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
27

---

For existing datasets that are re-packaged, both the original license and the license of the derived asset (if has changed) should be provided.
If this information is not available online, the authors are encouraged to reach out to the asset's s creators.
13. New Assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [Yes]
Justification: Our code is provided supplement.
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details the dataset/code/moodl as part their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset used.
At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsoureing and Research with Human Subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well details about compensation (if any)?
Answer: [NA]
Justification: This paper does involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Including this information the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included the main paper.
According the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or equivalent approval/review based the requirements of your country or institution) were obtained?
Answer: [No]
Justification: This paper does not involve crowdsourcing research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Depending the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.
28


Reviews:
[{"review_id": "nGWxkgi8A9", "reviewer": "Reviewer_vBXw", "summary": "This paper learns a generic representation of electricity time series data. The proposed PowerPM model is composed of a temporal encoder and a hierarchical encoder.", "questions": "Can the authors elaborate the settings of the freeze version?\n\nWhat are the effects of contrastive learning?\n\nThe figure 4 looks like a software snapshot. Can the authors explain the major information conveyed in this figure?", "limitations": "The paper describes techniques limited to energy forecasting tasks, and it is not clear if the methods can be applied to other tasks.", "rating": 3, "confidence": 3, "soundness": 2, "presentation": 2, "contribution": 1, "strengths": "The results shown in the table exhibit good numerical results of the proposed model.", "weaknesses": "It is unclear where the performance gain comes from. And the proposed architecture is a combination of several previous techniques, so it is hard to identify the technical contributions.\n\nMoreover, it is not well motivated on the usage of developed models for many power system tasks. As far as the reviewer is aware of, current statistical methods and machine learning-based methods can already give good load forecasting results. The real challenges come from noisy data inputs, incomplete number of features, or small-region level forecasting. Yet this paper only looks into a very general forecasting problem, and does not reveal the key challenges and special properties of energy forecasting.\n\nThe grid stability notion is quite misleading. In standard power system tasks, stability notion is with respect to either frequency or voltage, and is more related to the transient states of the systems. While in this paper, the stability is more like a phenomenon or behavior analysis. \n\nIn addition, the paper needs to discuss the computational costs related to both training and inference, as utility and grid operators normally are not equipped with enough computation capabilities to use large foundation models.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your constructive and detailed comments. We apologize for the imprecise claims in certain parts of our manuscript. We will reconsider the claims to be more precise. We hope that the responses below could address your specific comments.\n\n**1. W1: Clarify where the performance of the model comes from and the technical contribution of the model** \n\nThank you for your valuable comments. Please refer to  **Global Response** for answers to these questions.\n\n**2. W2&L1: Only looks into a very general forecasting problem and disscuss generalization**\n\nMost of methods forecast on a single or small number of instances and have poor generalization. Due to the large difference in the electricity consumption of different users, the effect decreases significantly when it transfers to other regions. Secondly, most studies do not consider the correlation between hierarchies in ETS modeling. Our experimental results show that PowerPM achieves SOTA on the four public datasets, demonstrating strong generalization of the forecasting task and modeling hierarchical relationships can improve the effect of forecasting tasks.\n\nAt the same time, **our model is not limited to forecasting tasks**, but can also perform tasks such as missing value imputation, electricity theft detection and so on. Through large-scale pre-training, PowerPM can successfully generalize to 44 power system tasks, and has a strong generalization ability, which is fundamentally different from the current end-to-end developed models for many power system tasks\n\n**3. W3: The grid stability notion is misleading**\n\nAs you said, stability notion is with respect to either frequency or voltage, and  transient states of the system. And these tasks are also summarized around these aspects:\n\n$\\bullet$ Electricity imputation:The missing load value will lead to misleading scheduling decisions, which will lead to the instability of the system voltage and destroy the grid stability[R1]. \\\n$\\bullet$ Clock anomaly detection:Clock anomalies will lead to clock frequency instability, resulting in synchronization errors, affecting the coordinated operation of power grid equipment, which will lead to instantaneous state out of control, affecting the grid stability[R2].\\\n$\\bullet$ Electricity theft detection: The behavior of stealing electricity will lead to the voltage change of local power grid, affecting the normal consumption of other users, which will also affect the grid stability[R3].\n\nSpecifically, these three tasks are all about the frequency or voltage and the itransient states of the system, so we summarize them as grid stability.\n\n**4. W4: Disscuss on computational costs**\n\nThank you for your valuable comments. Please refer to  **Global Response** for answers to these questions.\n\n**5. Q1: The effects of contrastive learning**\n\nContrastive learning is used to capture the differences between different ETS windows. The results of Ablation experiments can be referred to in section 3.4 \"Ablation Study\", which shows that contrastive learning can effectively improve the performance of downstream tasks related to classification and anomaly detection. the related literature points that contrastive learning has assumed the downstream applications to be classifications[R4].\n\n**6. Q2: Elaborate the settings of the freeze version**\n\nWe are sorry that we don't provide clear guidance in the manuscript so that you missed this part. The details of frozen version of PowerPM can be find in Appendix D.1, \"Partial Fine-tuning\" section.\n\nIn the P-FT (Partial Fine-tuning) setup, for different tasks, we also introduce different head $H$ on the top of pre-trained encoder $f(.)$. \nFor forecasting and imputation tasks, we use a prediction $H_l$  head to map prediction points or reconstruction points from $\\mathbf{z}_i$. \nAnd for anomaly detection and classfication tasks, a classifier $H_c$ on top of the pre-trained encoder $f(.)$.  \nDuring the whole finetune process, we keep the parameters of $f(.)$ fixed. Only the head is fine-tuned in this setup.\n\n**7. Q3: Explain the major information conveyed in this Figure 4**\n\nAs you can see from the figure, the yellow bar chart indicates the electricity loss from actual orderly reductions  in electricity use, whereas the blue bar chart represents the electricity loss from scheduled  orderly reductions in electricity use. As observed from the figure, the scheduled orderly reductions predicted by PowerPM approach the energy loss resulting from actual reductions, which is sufficient to provide guidance for the planning of power systems, demonstrating the effectiveness of PowerPM. A detailed description can be found in Appendix B.\n\nIf you still have any other new concerns, we would be eager to know what we can do to address any questions or concerns you may have. Thanks for your thoughtful feedback again!\n\nReference:\n\n[R1] Wang M C, Tsai C F, Lin W C. Towards missing electric power data imputation for energy management systems[J]. Expert Systems with Applications, 2021, 174: 114743.\n\n[R2] Zhang H, Wang Q, Li Y, et al. Clock Anomaly Detection Method of Power Quality Monitoring Device Based on Voltage Sag[C]//2021 IEEE 2nd China International Youth Conference on Electrical Engineering (CIYCEE). IEEE, 2021: 1-6.\n\n[R3] Depuru S S S R, Wang L, Devabhaktuni V. Electricity theft: Overview, issues, prevention and a smart meter based approach to control theft[J]. Energy policy, 2011, 39(2): 1007-1015.\n\n[R4] Liu X, Zhang F, Hou Z, et al. Self-supervised learning: Generative or contrastive[J]. IEEE transactions on knowledge and data engineering, 2021, 35(1): 857-876.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "G5wkAm3HrJ", "reviewer": "Reviewer_Q8uG", "summary": "This paper proposed a foundation model, PowerPM, to model electricity time series data, providing a large-scale off-the-shelf model for power systems. PowerPM consists of a temporal encoder and a hierarchical encoder with a self-supervised pretraining framework. The authors have tested PowerPM in Demand-side Management, Grid Stability, and Consumer Behavior Analysis, showing their advantages against other common time series LLM models, such as GPT4, TimeLLM, etc.", "questions": "1. For power systems, have you considered the power systems constraints, such as power network structures, physic laws of Kirchhoff's circuit laws?\n2. With renewable energy sources, have you considered of including weather time series data along with ETS data?", "limitations": "The authors has discussed the limitations of their work.", "rating": 7, "confidence": 4, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "1. They have tested their work with real data and showing competitive results;\n2. It is the first model that considers temporal dependency and hierarchical dependency simultaneously;\n3. Comprehensive model analyses.", "weaknesses": "The writing should be improved, several places are missing space between words.\nIt could be improved or make the paper more convincing if the author could compare their framework with some state-of-art techniques. However, it is understandable that the research question/application objectives are more comprehensive in this paper.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for all the insightful comments. We apologize for the imprecise claims in certain parts of our manuscript and misunderstandings caused by our writing. Responses to specific comments are listed below.\n\n**1. W1: Several places are missing space between words**\n\nThank you for pointing out our writing problems. We are sorry that we have caused you discomfort in reading due to our negligence. We've gone over The spelling issues and missing space, The updates are as follows:\n\n$\\bullet$ in line 77 of the manuscript, \"Model **(PowerPM),**\" will be changed into \"Model **(PowerPM) ,**\"\\\n$\\bullet$ in line 77 of the manuscript, ”about **250Mparameters**“ will be changed into ”about **250M parameters**“.\\\n$\\bullet$ in line 763 of the manuscript, \"pre-train on **Load** data.\" will be changed into \" pre-train on **load** data.\"\\\n$\\bullet$ in the caption of figure 1, \"(d)  Various tasks in power **systems**\" will be changed into \"(d)  Various tasks in power **systems.**\"\n\n**2. W2: Compare their framework with some state-of-art techniques**\n\nThank you for your advice. Since ETS is essentially a time series, we choose the time series modeling model as the baseline, and all of its models are tested in the field of power systems in the original article. \n\nOur baseline includes not only the SOTA model of supervised learning, but also the SOTA model of self-supervised learning. With the application of large language models in time series, we also select three LLM enhanced Time series models of SOTA. See section \"Experiment 3.1\" and Appendix 4.1 for a description of baselines.\n\nTherefore, our experiment included three categories of SOTA models, and conducted sufficiently complete experiments, as shown in Table 1.\n\n**3. Q1: Whether considered the power systems constraints**\n\nThank you for your suggestion, we are sorry that we did not take into account the  power systems constraints, such as Kirchhoff's circuit laws, etc. \n\nBecause all the data we currently obtain is provided by State Grid Corporation of China in Zhejiang province, it is the electricity consumption data after desensitization, which only contains the instance data recorded in the terminal meter and sensor. Its minimum granularity only sinks to the user unit and does not include complex circuit topological relationships within or between users. Therefore, we do not consider the constraints of the power network topology here, and only model ETS according to the physical level constraints in the real world, which can achieve quite superior results. \n\nBut it's worth noting, your suggestion is very promising and meaningful, because considering more constraints can better model the power system. In the future, we will consider including power network structures, such asKirchhoff's circuit laws and other factors to model ETS. Please keep paying attention to our work and thank you again for your suggestions.\n\n**4. Q2: Whether considered of including weather time series data**\n\nYes, as you said, we have considered weather time series data. Please refer to lines 135-142 in the \"Method\" section for the detailed description and figure 2(b) for the legend. \n\nWe crawled the local weather and temperature on the public website, called as exogenous variables, and assigned the learnable parameters according to its different values and mapped to the embedding table. Each ETS windows has a corresponding sequence of exogenous variables. When the temporal encoder models ETS, the corresponding exogenous variables will look up the corresponding representation in the embedding table index, and finally add and fuse to obtain the ETS representation enhanced by external variables.\n\nThank you again for your thorough consideration. The ablation experiment results are shown in Table 7 in the Appendix. After adding the weather and other exogenous variables, the performance of the model will indeed be improved to a certain extent, especially the Solar generation forecasting which is greatly affected by environmental factors.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "mCePv30rXC", "reviewer": "Reviewer_W2Rh", "summary": "A pre-trained LLM named PowerPM is proposed for modeling Electricity Time Series (ETS) data. PowerPM combines a temporal encoder for capturing temporal patterns and a hierarchical encoder for understanding hierarchical correlations. PowerPM also employs a self-supervised pre-training strategy that incorporates masked ETS modeling and dual-view contrastive learning, enhancing the model's ability to learn from the intricacies of ETS data. \n\nPowerPM is specifically designed for power systems, especially for the demand side, making it unique among other general models. This paper claims that deployment of PowerPM in Zhejiang Power Grid had led to significant economic benefits, showing the practical value of the model in real-world settings.", "questions": "The writing lacks sufficient detail and explanation, resulting in quite some questions.\nFor example, what are the computational resources used? The provided materials do not include codes related to the proposed pre-training method. No information about whether the pre-trained model or the pre-training dataset will be released to make the work transparent.", "limitations": "This works involves load forecasting for power system and individual users in real-world applications, which would affect power system operation, its safety, and societal impact as a whole. These should be discussed.", "rating": 3, "confidence": 4, "soundness": 3, "presentation": 2, "contribution": 2, "strengths": "This article proposes a model for load time series forecasting on multi-hierarchy demand side and pre-trains it on a large amount of real data. It shows good performance and strong generalizability in experiments. Furthermore, the model is applicable in the real world.\nFor multiple pre-training challenges on multi-hierarchy load time series data, this method integrates corresponding solutions together.", "weaknesses": "Since the pre-training dataset is comprehensive, the work lacks an analysis on the effectiveness provided by the dataset. In other words, readers may not understand if the dataset works a lot or the proposed method.\nThe writing and explanation in this paper need to be improved.\nThis paper involves extensive training, but details are largely missing from the current paper.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your constructive and detailed comments. Responses to specific comments are listed below.\n\n**1. W1: About source of the model performance**\n\nThank you for pointing out that our description may confuse the reader. We have made a detailed reply in **Global Response** , please check and wish it can answer your confusion.\n\n**2. Q1: About computational resources used**\n\nWe are sorry that we don't provide clear guidance in the manuscript so that you missed this part. The computational costs and training process can be found in Appendix D.1. All the experiments are repeated five times, conducted on a Linux system with 2 CPUs (AMD EPYC 9654 96-Core Processor) and 8 GPUs (NVIDIA Tesla A800 80G) for about 8 days.\n\n**3. Q2: Codes related to the proposed pre-training method**\n\nSorry we didn't have a clear description of the submitted code, so that you missed this pre-training method.\n\nThe complete code has been placed in the supplementary material and has been uploaded at the time of the first submission of the manuscript. The file structure is as follows: (Hidden files have been ignored)\n\n.\\\n├── configs\\\n├── environment.yaml\\\n├── logs\\\n├── Makefile\\\n├── pyproject.toml\\\n├── result\\\n├── scripts\\\n│   └── pretrain\\\n│       └── run.sh\\\n├── src\\\n│   ├── models\\\n│   │   ├── powergpt_module_pretrain.py\\\n│   │   └── PowerGPT.py\\\n│   └── powergpt_pretrain.py\\\n└── tests\n\nThe entrance is in the process of the pretraining is **./ scripts/pretrain/run.sh**,  the script will execute python **./src/powergpt_pretrain. py**. Meanwhile, the **./src/models/powergpt_module_pretrain.py** file contains the pre-training process, where **loss_cl** and **loss_mse** are the loss functions corresponding to the two pretraining tasks.\nAnd **./src/models/PowerGPT.py** is the code file of our model backbone, which contains all the implementation code for the model and the detailed procedure for the pre-training tasks. That's all the relevant code and files for our pre-training.\n\n**4. Q3: Our planned steps for the dataset and pre-trained model release**\n\nWe apologize for the temporary unavailability of the dataset, because it involves highly sensitive user electricity usage data and personal privacy concerns. But we will release our pre-trained model.\n\nIn the future, in order to make this dataset a scientific research tool and better serve the research community, we plan to promote the release of our dataset in the following three stages:\n\n$\\bullet$ Stage1: We plan to publicly release the pre-trained models with different scale in 250M、128M、64M and 35M, shortly after our work is accepted. And we also release the four public datasets. This will enable other researchers to not only replicate our model’s experimental results, but also utilize these models for other research of interest.\n\n$\\bullet$ Stage2: We will actively communicate with the State Grid Corporation of China in Zhejiang province and aim to release the raw data of a portion of instances by the end of the year, to support further research efforts. The same as the work in our manuscript, these releases will be also conducted in compliance with ethical review requirements.\n\n$\\bullet$ Stage3: In the future, we will explore the possibility of releasing the full dataset following approval of the relevant ethical review, to allow researchers to use the large-scale dataset for more research.\n\n**5. L1: Ethics statement**\n\nThe data collection and experiments conducted in our work have been approved by the Institutional Review Board (IRB) and passed ethical review. The data has been effectively licensed by State Grid Corporation of China in Zhejiang province, all user-related information is encrypted, and all downstream tasks are only used for State Grid supply planning and internal analysis.\n\n\nIf you still have any other new concerns, we would be eager to know what we can do to address any questions or concerns you may have. Thanks for your thoughtful feedback again!", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "qbfcjHSrHr", "reviewer": "Reviewer_Vuy7", "summary": "The paper introduces the PowerPM: Foundation Model for Power Systems, which is designed to address the challenges of learning a generic representation of electricity time series (ETS) data in power systems. The model incorporates a temporal encoder and a hierarchical encoder to effectively capture the complex hierarchical structure and temporal dependencies present in ETS data. Additionally, PowerPM utilizes a self-supervised pre-training framework that includes masked ETS modeling and dual-view contrastive learning to enhance its ability to capture temporal dependencies and discrepancies across ETS windows. Overall, the contributions of the paper lie in the development of a sophisticated model that can accurately represent and analyze ETS data in power systems, offering valuable insights for various applications in the field.", "questions": "### Questions and Suggestions for the Authors:\n\n1. **Clarification on Model Interpretability:**\n   - Can the authors provide more insights into how the PowerPM model interprets and captures temporal dependencies and hierarchical correlations in electricity time series data? Understanding the interpretability of the model can enhance its transparency and trustworthiness in real-world applications.\n\n2. **Explanation on Hyperparameter Selection:**\n   - Could the authors elaborate on the rationale behind the selection of specific hyperparameters for the PowerPM model? Providing details on the hyperparameter tuning process and its impact on model performance would offer valuable insights into the model's robustness.\n\n3. **Discussion on Generalization to Unseen Domains:**\n   - How does the PowerPM model generalize to unseen domains or datasets outside the ones used in the experiments? Understanding the model's ability to adapt to new data distributions and scenarios is crucial for assessing its practical utility in diverse real-world applications.\n\n4. **Scalability and Deployment Considerations:**\n   - What are the scalability considerations for deploying the PowerPM model in large-scale power systems? How does the model handle real-time data processing and what are the potential challenges in scaling it for industrial applications?\n\n5. **Ethical and Fairness Implications:**\n   - Have the authors considered the ethical implications of using the PowerPM model in power systems, particularly in terms of data privacy, fairness, and potential biases? Addressing these concerns can ensure responsible and unbiased deployment of the model in practice.\n\n6. **Future Directions and Extensions:**\n   - Are there any plans to extend the PowerPM model or explore new research directions based on the current findings? Discussing potential future developments or applications of the model can provide insights into the ongoing research trajectory in the field of power systems modeling.\n\n7. **Comparison with Latest Research:**\n   - Have the authors considered comparing the PowerPM model with the most recent state-of-the-art models in electricity time series analysis? Including such comparisons can strengthen the paper's contribution and highlight the advancements made by PowerPM in the field.\n\nAddressing these questions and suggestions can enhance the clarity, robustness, and applicability of the PowerPM model, providing valuable insights for both the research community and practical stakeholders in the power systems domain.", "limitations": "### Assessment of Limitations and Societal Impact Considerations:\n\n1. **Limitations Addressed:**\n   - The authors have discussed the limitations of their work in Appendix F, which is a positive step towards transparency and acknowledging the constraints of the research. This demonstrates a willingness to reflect on the scope and potential weaknesses of the study.\n\n2. **Societal Impact Considerations:**\n   - The paper lacks a comprehensive discussion on the potential negative societal impacts of deploying the PowerPM model in real-world power systems. While some ethical considerations are mentioned, a more thorough exploration of broader societal implications is needed.\n\n### Suggestions for Improvement:\n\n1. **Enhanced Limitations Section:**\n   - The authors could consider expanding the limitations section in the main body of the paper rather than relegating it to an appendix. This would ensure that readers engage with the limitations more prominently and understand the boundaries of the research.\n\n2. **Broader Societal Impact Analysis:**\n   - To address potential negative societal impacts, the authors should conduct a detailed analysis of how the deployment of the PowerPM model could affect various stakeholders, including issues related to fairness, privacy, and bias. Providing mitigation strategies for these impacts would demonstrate a proactive approach to responsible research.\n\n3. **Incorporate Ethical Frameworks:**\n   - Utilize established ethical frameworks or guidelines to evaluate the ethical implications of the PowerPM model in power systems. This could involve considering principles such as fairness, accountability, transparency, and privacy to ensure ethical deployment and minimize negative societal consequences.\n\n4. **Engage with Stakeholders:**\n   - Engage with relevant stakeholders, such as power system operators, regulators, and community representatives, to gather diverse perspectives on the potential societal impacts of implementing the PowerPM model. Incorporating feedback from stakeholders can enrich the discussion on societal implications.\n\n5. **Mitigation Strategies:**\n   - Propose concrete mitigation strategies for addressing any identified negative societal impacts of the PowerPM model. This could include mechanisms for monitoring model performance, ensuring fairness in decision-making processes, and enhancing transparency in model deployment.\n\nBy addressing these suggestions, the authors can strengthen the ethical foundation of their research, demonstrate a commitment to responsible innovation, and contribute to a more comprehensive understanding of the societal implications of deploying advanced models like PowerPM in practical settings.", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "### Strengths Assessment:\n\n1. **Originality:**\n   - The paper demonstrates originality in its approach to addressing the challenges of modeling electricity time series data in power systems. The combination of a temporal encoder and hierarchical encoder, along with the self-supervised pre-training framework, showcases innovative thinking in capturing temporal dependencies and hierarchical correlations simultaneously.\n   - The use of masked ETS modeling and dual-view contrastive learning in the pre-training stage adds a novel dimension to the model's ability to learn universal representations from ETS data.\n\n2. **Quality:**\n   - The quality of the paper is evident in the thorough description of the PowerPM model, its components, and the experimental results obtained. The model's deployment in real-world scenarios and the achievement of state-of-the-art performance on diverse downstream tasks within private and public datasets reflect the high quality of the research.\n   - The incorporation of extensive experiments, ablation studies, and few-shot experiments provides a robust evaluation of the model's effectiveness and generalization ability across various tasks and domains.\n\n3. **Clarity:**\n   - The paper is well-structured and clearly articulates the motivation, methodology, and results of the PowerPM model. The descriptions of the temporal encoder, hierarchical encoder, and the self-supervised pre-training framework are presented in a coherent manner, making it easy for readers to understand the technical aspects of the model.\n   - The inclusion of figures, tables, and detailed explanations aids in clarifying complex concepts such as the model analysis, ablation study, and model scale evaluation.\n\n4. **Significance:**\n   - The significance of the paper lies in its contribution to advancing the modeling of electricity time series data in power systems. By introducing the PowerPM model, which effectively captures temporal dependencies and hierarchical correlations, the research offers a valuable tool for enhancing economic efficiency and promoting low-carbon principles in power systems.\n   - The model's superior performance on diverse downstream tasks, its generalization ability across different datasets, and the economic benefits generated in real-world deployments underscore the practical significance of the research in the field of power systems modeling.\n\nOverall, the paper excels in originality, quality, clarity, and significance, making a substantial contribution to the domain of electricity time series data modeling in power systems.", "weaknesses": "### Weaknesses Assessment:\n\n1. **Limited Comparison with State-of-the-Art Models:**\n   - While the paper highlights the superior performance of PowerPM over baseline models, there is a lack of comparison with the most recent state-of-the-art models in the field of electricity time series data analysis. Including comparisons with cutting-edge models could provide a more comprehensive evaluation of PowerPM's performance [T1].\n\n2. **Insufficient Discussion on Model Interpretability:**\n   - The paper could benefit from a more in-depth discussion on the interpretability of the PowerPM model. Providing insights into how the model captures and represents temporal dependencies and hierarchical correlations in ETS data could enhance the understanding of its inner workings and decision-making processes.\n\n3. **Limited Exploration of Hyperparameters and Sensitivity Analysis:**\n   - The paper lacks a detailed exploration of hyperparameters and sensitivity analysis for the PowerPM model. Including a thorough investigation of the impact of hyperparameters on model performance and conducting sensitivity analysis could provide valuable insights into the robustness and stability of the model across different settings.\n\n4. **Scalability and Efficiency Considerations:**\n   - The paper could further address scalability and efficiency considerations of the PowerPM model, especially when applied to large-scale datasets or real-time applications. Discussing the computational requirements, training time, and potential bottlenecks in scaling the model could help in understanding its practical feasibility in industrial settings.\n\n5. **Limited Discussion on Ethical and Societal Implications:**\n   - The paper could expand its discussion to include ethical and societal implications of deploying the PowerPM model in real-world power systems. Addressing issues related to data privacy, fairness, and potential biases in the model's predictions could enhance the overall impact and relevance of the research in a broader context.\n\n### Suggestions for Improvement:\n\n1. **Incorporate Comparison with State-of-the-Art Models:**\n   - Conduct a thorough comparison with the latest state-of-the-art models in electricity time series analysis to provide a more comprehensive evaluation of PowerPM's performance and highlight its competitive advantages.\n\n2. **Enhance Model Interpretability Analysis:**\n   - Include a section dedicated to explaining how the PowerPM model interprets and processes ETS data, shedding light on its decision-making processes and enhancing the transparency of the model.\n\n3. **Conduct Hyperparameter Tuning and Sensitivity Analysis:**\n   - Perform a detailed hyperparameter tuning process and sensitivity analysis to understand the impact of key parameters on model performance and ensure robustness across different scenarios.\n\n4. **Address Scalability and Efficiency Concerns:**\n   - Discuss the scalability and efficiency aspects of the PowerPM model, including computational requirements, training time optimization strategies, and considerations for real-time deployment in power systems.\n\n5. **Expand Discussion on Ethical and Societal Implications:**\n   - Include a section on the ethical and societal implications of deploying the PowerPM model, addressing issues of fairness, bias, and privacy to ensure responsible and ethical use of the model in practical applications.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your constructive and detailed comments. Responses to specific comments are listed below.\n\n**1. W1&Q7: Compare their framework with some state-of-art techniques**\n\nThank you for your advice. Since ETS is essentially a time series, we choose the time series modeling model as the baseline, and all of its models are tested in the field of power systems in the original article. \n\nOur baseline includes not only the SOTA model of supervised learning, but also the SOTA model of self-supervised learning. With the application of large language models in time series, we also select three LLM enhanced Time series models of SOTA. See section \"Experiment 3.1\" and Appendix 4.1 for a description of baselines.\n\nTherefore, our experiment included three categories of SOTA models, and conducted sufficiently complete experiments, as shown in Table 1.\n\n**2. W2&Q1: Discussion on model interpretability**\n\nThank you for your insightful comment on the interpretability of the PowerPM model. We acknowledge the importance of model interpretability and its impact on user trust and model adoption. Your feedback is valuable in guiding us to enhance the discussion on this aspect.\n\nIn terms of model architecture, PowerPM consists of a temporal encoder and a hierarchical encoder, whose design goals are clear and public. Time encoders utilize Transformer encoders to capture temporal dependencies in time series and learn to predict future time points through mask ETS modeling strategies in the self-supervised learning framework.\n\nHierarchical encoders use graph neural network technology, such as R-GCN, to simulate complex relationships between different hierarchy. This representation of hierarchical correlations is essential for understanding power system dynamics from both macro and micro perspectives.\n\nOur self-supervised learning framework further enhances the model's understanding of temporal dependence and differences across ETS windows. This not only improves the performance of the model, but also gives us a way to explain how the model learns these dependencies.\n\n**3. W3&Q2: Explanation on Hyperparameter Selection**\n\nThank you for pointing out our problem. Due to the large number of parameters and huge pre-training data, and the long pretrain time of the model, considering that the existing foundation model work has not carried out the super-parameter search, for the sake of efficiency, the super-parameter in the pre-training stage is set by default with reference to the existing work.\n\n**3. W4&Q4: Scalability and Deployment Considerations**\n\nThank you for raising these doubts. We have successfully deployed a lightweight version of PowerPM on the national grid for load forecasting.\n\nThe main challenge encountered during the deployment process is that the inference resources of the National Grid are limited, resulting in low efficiency of the model operation and can not get real-time results. The solution is as follows:\n\nDuring the deployment phase, we trained models with different parameters for different tasks, including 120M and 250M, to achieve a balance of efficiency and accuracy. It is deployed on the grid's six NVIDIA 3090 for inference. For functions that are called more frequently, such as user load forecasting, we take into account efficiency factors and use a 120M model for inference. For functions that are called less frequently, such as area load forecasting, a 250M model can be used for inference, thus guaranteeing accuracy.\n\nAt the same time, this scalability is very free, and the architecture of the model can be modified according to the number of parameters you want to adjust.\n\n**4. W5&Q5&L2: Ethics statement and social impact**\n\nThe data collection and experiments conducted in our work have been approved by the Institutional Review Board (IRB) and passed ethical review. The data has been effectively licensed by State Grid Corporation of China in Zhejiang province, all user-related information is encrypted, and all downstream tasks are only used for State Grid supply planning and internal analysis.\nThe social impact is shown as Appendix G.\n\n**5. Q3: Discussion on Generalization to Unseen Domains**\n\nThank you for asking this question. Since PowerPM is a foundation model proposed for the power system, it cannot be generalized to data sets in the unknown field used in the experiment, but it can be generalized to any data set in the power system, which has been verified in the paper. We conducted training on private data sets and generalized it to four public datasets. The result is shown in Table 2.\n\nIf the data set has a hierarchical relationship, the graph structure can be constructed. If there is no hierarchy (that is, the graph structure is an isolated graph), PowerPM can model normally without any modification. Therefore, PowerPM has a strong ability to adapt to new data distributions and scenarios in power system.\n\n**6. Q6: Future Directions and Extensions**\n\nThank you for your interest in the future direction of our work. The future development of PowerPM is divided into the following two stages:\n\n$\\bullet$ Stage 1: Deploy as many downstream tasks as possible to the National Grid for landing. This will make grid data analysis more intelligent and automated, and assist grid workers in decision-making, reducing costs and increasing efficiency.\\\n$\\bullet$ Stage 2: Currently, we only consider the macro-level interaction modeling. In the future, we will consider more factors, such as power grid topology, Kirchhoff's law and other constraints. To enhance the modeling process of ETS data to achieve a more general representation. This enhances downstream task performance and provides a more unique perspective for modeling ETS.\n\nAbove is the future development direction of our work, thank you for your attention and support to our work.\n\nIf you have any other concerns, please let us know what we can do to address any concerns you may have. Thanks for your thoughtful feedback again!", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:
## **Global Response to AC and all reviewers** ##
## About source of the model performance  ##
Thanks to all reviewers for the careful reading and thoughtful feedback. Here we explain the effectiveness of the dataset and the source of the model performance, as a solution to similar concerns raised by some reviewers. These concerns include:

$\bullet$ it is unclear where the performance gain comes from.(vBXw) \
$\bullet$ the lack of an analysis on the dataset effectiveness. Not understand if the dataset works a lot or the proposed method.(W2Rh)

**1. the effectiveness of dataset**

We regret that the following fact is not clearly explained in the manuscript: the pretrain dataset used in PowerPM is provided by State Grid Corporation of China in Zhejiang province, whose **authenticity** and **effectiveness** can be guaranteed. For a detailed description, please see Appendix C.

**2. source of the model performance**

We construct hierarchies for the dataset and introduce external weather variables, this is a feature of the dataset that was not included in previous work and is beneficial for ETS modeling. However, it is important to note that: to ensure the fairness of our experiments, all pretraining based baselines are re-pretrained on our pretraining dataset, instead of utilizing the weights published (if any) by the original work. In other words, all the performance comparison are conducted on the same data and with the similar schemes. We make fair comparisons not only on private datasets, but also on public datasets. The experimental results are shown in the Table 1 and Table 2.

Therefore, the performance improvement is mainly due to our superior models, which include temporal encoder and hierarchical encoder, and well-designed pre-training tasks, which can better utilize the characteristics of ETS. The advantages of each module can be highlighted in the ablation experiment, as shown in " 3.4 Model Analysis" section.

Therefore, the performance gains are due to our better model. Datasets play only a small part.

## About technical contribution ##
Above, we explain the source of the model performance, and here we summary the technical contribution to enable the reviewer to have a clearer understanding, And response the concern : The technical contribution is unclear.(vBXw)

The novelty of the proposed method are listed below:

$\bullet$ Self-supervised pre-training tasks:

Our method contains two novel self-supervised pretraining tasks(masked ETS modeling and dual-view contrastive learning), considering the characterics of ETS. The motivation of our designed tasks is to keep the  temporal dependency within continuous ETS windows and preserve the unique patterns of across different instance, which is more consistent with the real electricity consumption pattern.

$\bullet$ Model design:

$\circ$ In temporal encoder, we first propose to integrate exogenous variables with embedding table for ETS modeling. Different from other multi-variable time series modeling methods[40, 49, 44], our exogenous variables series are not input as ETS features, but are mapped as trainable parameters, which improves the flexibility and accuracy of ETS modeling.

$\circ$ In hierarchical encoder, we first propose the use of the real physical hierarchy to enhance ETS modeling. Different from other time series modeling methods with strong hierarchical constraints[R1-R3], we fully consider the characteristics of ETS data, propose to build hierarchical graph using region relationships, and use GNN to model these hierarchical correlation for the first time.

Benefit from the above technical contribution, we can achieve better performance in this scenario.

## About computational costs ##
We will explain the computational costs during train, inference and deployment to response the concern of reviewer vBXw,  W2Rh and Vuy7:

We are sorry that we don't provide clear guidance in the manuscript so that you missed this part. The computational costs and training process can be found in Appendix D.1.
All the experiments are repeated five times, conducted on a Linux  system with 2 CPUs (AMD EPYC 9654 96-Core Processor) and 8 GPUs (NVIDIA Tesla A800 80G)  for about 8 days.

We use a single NVIDIA 3090 for inference. According to different tasks, the inference time is about 1h for users' weekly load forecasting and 2s for city load forecasting.

During the deployment phase, we trained models with different parameters for different tasks, including 120M and 250M, to achieve a balance of efficiency and accuracy. It is deployed on the grid's six NVIDIA 3090 for inference.
For functions that are called more frequently, such as user load forecasting, we take into account efficiency factors and use a 120M model for inference.
For functions that are called less frequently, such as area load forecasting, a 250M model can be used for inference, thus guaranteeing accuracy.

## About Specific Responses ##

We have individually addressed all of your comments below, specifically addressing each reviewer’s concerns in the corresponding responses. Please note that in our responses, references in the format "[R1]" indicate citations that are newly added in the rebuttal, while references in the format "[1]" are citations from the original manuscript.

We have dedicated significant effort to improving our manuscript, and we sincerely hope that our responses will be informative and valuable. We would love to receive your further feedback.

Reference：

[R1] Orcutt G H, Watts H W, Edwards J B. Data aggregation and information loss[J].  The American Economic Review, 1968, 58(4): 773–787.

[R2] Anderer M, Li F. Hierarchical forecasting with a top-down alignment of independent-level forecasts[J]. International Journal of Forecasting, 2022, 38(4): 1405-1414.

[R3] Pang Y, Yao B, Zhou X, et al. Hierarchical Electricity Time Series Forecasting for Integrating Consumption Patterns Analysis and Aggregation Consistency[C]//IJCAI. 2018: 3506-3512.

Meta-review: The paper proposes a pre-trained model for electricity time series data, combining temporal and hierarchical encoders to capture complex dependencies in power systems. Initial reviews were mixed, acknowledging comprehensive experiments and potential practical impact, but raising concerns about the source of performance gains, computational costs, and comparisons with state-of-the-art methods.

In their rebuttal, the authors provided detailed responses, clarifying performance gains, computational costs, and ethical considerations. They also outlined plans for model and dataset release. 

Two reviewers who recommended rejection did not respond to the rebuttal despite repeated reminders. Their concerns were addressed by the rebuttal. The paper also went through extensive ethical reviews. The concerns of the ethical reviewers were also addressed. Therefore I tend to side with the two positive reviewers and recommend to accept this submission due to its solid methodology and especially its potential practical impact.

I read the paper myself and am impressed by its practical real applications. Due to the fact that I overruled the two negative but non-responsive reviewers, I discussed with the senior AC about this paper. The SAC agreed with my decision.
Decision: Accept (poster)