Title: PowerPM: Foundation Model for Power Systems
Venue: NeurIPS 2024
Accepted: yes
Keywords: time series pre-training

power systems

foundation model
Abstract: The proliferation of abundant electricity time series (ETS) data presents numerous opportunities for various applications within power systems, including demand-side management, grid stability, and consumer behavior analysis. Deep learning models have advanced ETS modeling by effectively capturing sequence dependence. However, learning a generic representation of ETS data for various applications is challenging due to the inherently complex hierarchical structure of ETS data. Moreover, ETS data exhibits intricate temporal dependencies and is susceptible to the influence of exogenous variables. Furthermore, different instances exhibit diverse electricity consumption behavior. In this paper, we propose a foundation model PowerPM for ETS data, providing a large-scale, off-the-shelf model for power systems. PowerPM consists of a temporal encoder and a hierarchical encoder. The temporal encoder captures temporal dependencies within ETS data, taking into account exogenous variables. The hierarchical encoder models correlations between different levels of hierarchy. Furthermore, PowerPM leverages a novel self-supervised pre-training framework consisting of masked ETS modeling and dual-view contrastive learning. This framework enables PowerPM to capture temporal dependency within ETS windows and aware the discrepancy across ETS windows, providing two different perspectives to learn generic representation. Our experiments span five real-world scenario datasets, including both private and public data. Through pre-training on massive ETS data, PowerPM achieves SOTA
performance on diverse downstream tasks within the private dataset. Notably, when transferred to public datasets, PowerPM retains its edge, showcasing its remarkable generalization ability across various tasks and domains. Moreover, ablation studies and few-shot experiments further substantiate the effectiveness of our model.

PowerPM: Foundation Model for Power Systems

Abstract

The proliferation of abundant electricity time series (ETS) data presents numerous opportunities for various applications within power systems, including demand-side management, grid stability, and consumer behavior analysis. Deep learning models have advanced ETS modeling by effectively capturing sequence dependence. However, learning a generic representation of ETS data for various applications is challenging due to the inherently complex hierarchical structure of ETS data. Moreover, ETS data exhibits intricate temporal dependencies and is susceptible to the influence of exogenous variables. Furthermore, different instances exhibit diverse electricity consumption behavior. In this paper, we propose a foundation model PowerPM for ETS data, providing a large-scale, off-the-shelf model for power systems. PowerPM consists of a temporal encoder and a hierarchical encoder. The temporal encoder captures temporal dependencies within ETS data, taking into account exogenous variables. The hierarchical encoder models correlations between different levels of hierarchy. Furthermore, PowerPM leverages a novel self-supervised pre-training framework consisting of masked ETS modeling and dual-view contrastive learning . This framework enables PowerPM to capture temporal dependency within ETS windows and aware the discrepancy across ETS windows, providing two different perspectives to learn generic representation. Our experiments span five real-world scenario datasets, including both private and public data. Through pre-training on massive ETS data, PowerPM achieves SOTA performance on diverse downstream tasks within the private dataset. Notably, when transferred to public datasets, PowerPM retains its edge, showcasing its remarkable generalization ability across various tasks and domains. Moreover, ablation studies and few-shot experiments further substantiate the effectiveness of our model.

1 Introduction

The volume of Electricity Time Series (ETS) data has recently increased rapidly due to the emergence of advanced power systems known as smart grids [10]. This abundance of data has paved the way for diverse applications in power systems, including demand-side management [22], grid stability [2] and consumer behavior analysis [49], etc. Meanwhile, these applications have spawned various tasks, as shown in Fig. 1(d). These include load forecasting [27, 4], clock anomaly detection [46], electricity theft [15] and and the detection of elderly individuals living alone [45].

Jing Zhang

Disctrict

User

City

:interaction

:instance

(a)

(b)

(c)

(d)

exogenous

variables

06:00

12:00

18:00

24:00

00:00

...

Demand-side management

-Load forecasting

-Solar generation forecasting

Grid stability

-Load imputation

-Electricity theft detection

Consumer behavior analysis

-Elderly living alone detection

-High-power appliances detection

Total 44 tasks

A

B

day

week

ETS windows

macro

micro

...

As society progresses towards modernization, electricity consumption is rapidly increasing, presenting opportunities and challenges for the development and application of smart grids. On one hand, the substantial economic benefits that accompany this significant electricity usage are considerable. On the other hand, unreasonable electricity planning can have a detrimental impact on the environment[30]. Therefore, given the large volume of data and the variety of tasks, there is an urgent need to study effective ETS data modeling methods for these tasks, so as to improve economic efficiency while adhering to low-carbon principles.

Recently, numerous research studies on pre-training approaches for ETS data have emerged. These approaches adopt the 'pre-training then fine-tuning' paradigm to deal with the dilemma of limited annotation data, and the pre-trained model to easily adapt to new tasks, such as PatchTST [21], TS2Vec [42], CoST [37], etc. However, these pre-training methods only utilize small-scale of data with a small number of instances (e.g. users), resulting in poor performance on downstream tasks. As the same time, many researcher begin to apply Large Language Models (LLMs) to assist time series modeling by using pre-trained LLM to encode time series [51] or incorporating additional descriptions related to the time series [17, 20]. Nevertheless, these models have limited ability in the power system scenario due to insufficient pre-training data of power systems and the lack of sufficient domain-specific knowledge. Additionally, none of these models are tailored for the scenario of power systems, so they neglect the unique characteristics of ETS data. Consequently, there remains a significant research gap in existing power systems literature regarding the modeling of ETS data using a foundation model.

In our scenario, the ETS data contains numerous instances and naturally exhibits a complex hierarchy [41, 23]. As depicted in Fig. 1(a), a city ETS can be disaggregated into district ETS accroding to the administrative divisions, which can further be disaggregated into user ETS in this district. For the complex hierarchy of ETS data, modeling ETS data entails the consideration of several challenges:

Hierarchical Dependency Modeling. The hierarchy of ETS data facilitates information interaction across different granularities. Fine-grained ETS provides detailed insights into individual electricity usage, while coarse-grained ETS for districts and cities captures broader factors and indicates overall trends. For example, user-level data reflects user-specific behaviors and city-level data encompasses demographics and policy effects [29, 35]. Integrating these levels of granularity to provide both macro and micro perspectives is a complex task that requires sophisticated modeling.

Temporal Dependencies within ETS Window. An ETS window refers to a piece of electricity time series over a period of time. The temporal dependencies within an ETS window refer to the correlations and dependencies between observations at different timestamps. As shown in Fig. 1(b), the city-level ETS exhibits daily and weekly dependency. Moreover, the temporal dependencies are often influenced by exogenous variables, such as weather, temperature, and seasonal effects. Integrating these factors into the model is challenging because their impact may interact with the temporal dynamics in complex ways. Accurately capturing the temporal dependencies with the impact of exogenous variables is a key challenge in modeling ETS data.

Discrepancy across ETS Windows. The patterns observed in ETS windows can vary significantly across different instances and different timestamps. For instance, as shown in Fig. 1(c), residential electricity consumption ( User A ) reaches its peak in the mornings and evenings, used for lighting, appliances, and heating. However, electricity usage typically declines during the day because residents

Demand-side Management

Exclusive User

Forecasting(MSE)

Public User

Forecasting(MSE)

District

Forecasting(MSE)

City

Forecasting(MSE)

Solar Generation

Forecasting(MSE)

One Fits All

TimeLLM

PatchTST

CoST

TimesNet

PowerPM

Consumer Behavior Analysis

One Fits All

TimeLLM

PatchTST

CoST

TimesNet

PowerPM

Elderly Alone

Detection (F0.5)

Gender

Classification (Acc.)

Family Structure

Classification (Acc.)

High-power Appliance

Detection (F0.5)

Age

Classification (Acc.)

Grid Stability

One Fits All

TimeLLM

PatchTST

CoST

TimesNet

PowerPM

Exclusive User

Imputation (MSE)

Public User

Imputation (MSE)

Industry

Imputation (MSE)

City

Imputation (MSE)

Electricity Theft

Detection (F0.5)

Clock Anomaly

Detection (F0.5)

(a)

(b)

(c)

are generally absent, being engaged in work or education activities outside the home. Moreover, industries ( User B ) have high power demand during specific daytime periods for machinery and production lines, with lower load requirements during nighttime and weekends. These variations in behavior highlight the challenge of achieving consistency across ETS windows in personalized modeling.

To address these challenges, we propose a foundation model for power systems named P ower P retrained M odel (PowerPM), as illustrated in Figure 3. PowerPM contains about 250 Mparameters and is pre-trained on large-scale hierarchical ETS data with 987 . 42 GB. Specifically, we employ the 'pretraining then fine-tuning' paradigm to learn generic representations by pre-training on hierarchical ETS data and to unify various tasks by fine-tuning on downstream data. During pre-training stage, we propose a novel self-supervised pre-training framework consisting of masked ETS modeling and dual-view contrastive learning , which enables PowerPM to capture temporal dependency within ETS windows and aware the discrepancy across ETS windows, so as to provide two different perspectives to learn universal representations. PowerPM mainly consists of two modules, namely, temporal encoder and hierarchical encoder . The temporal encoder employs Transformer encoders to capture the temporal dependency in ETS data, and incorporates exogenous variables to make the modeling process more robust. Moreover, to model hierarchical dependency, hierarchical encoder utilizes R-GCN [25] to propagate information about the correlation between hierarchy. According to the message that passes through the hierarchies, the micro and macro information can effectively assist in modeling the ETS data. In summary, the main contributions of our work include:

We propose a foundation model for power systems named PowerPM, which is pre-trained on large-scale ETS data and provide an off-the-shelf model for power systems.

To the best of our knowledge, PowerPM is the first to date that considers temporal dependency and hierarchical dependency simultaneously. In addition, we present a novel self-supervised pre-training framework that combines masked ETS modeling and dual-view contrastive learning, enhancing the model's ability to learn temporal dependencies within ETS windows and aware the discrepancy across ETS windows.

Extensive experiments show that PowerPM generalizes well to 44 downstream tasks. Fig. 2 summarizes the results of all the downstream tasks, showing its great potential in ETS data modeling. Moreover, when transferred to the public dataset, PowerPM maintains its superiority, showcasing its remarkable generalization ability across various tasks and domains. Further analysis illustrates the effectiveness of PowerPM as well.

2 Methodology

Overview. As shown in the middle part of Fig. 3: Firstly, the hierarchical graph G is constructed according to the naturally existing hierarchical relationship of ETS data. The ETS windows in G and its corresponding exogenous variables are denoted as { x i } N i =1 and { o i } N i =1 , where N is the number of instances, x i ∈ R T w , o i ∈ R T w × K , and each instance ETS window spans T w time points starting at T a and ending at T b . Each time point has K kinds of exogenous variables. Our objective is to perform

+

+

Masked ETS modeling

Indexing

reconstruct

predict

Random Mask

Casual Mask

Positive Pair in Temporal-View

Negative Pair in Temporal-View

Negative Pair in Instance-View

Duel-view

Constrastive Learning

Embeddings Table

+

P

C

Cluster 1

Hierarchical Structure

in Reality

Hierarchical Graph

Exogenous Variables

Patched Window

Projection

Transformer

Encoder

Hierarchical Graph

Construction

Self-supervised

Pre-training Task

Temporal

Encoder

Hierarchical

Encoder

City

District

User

Cluster

R-GCN

District-User

District-Cluster

District-City

Cluster-District

City-District

Adjacency

Matrix

Patching

...

Exogenous

Variables

(c)

(d)

(a)

(b)

pre-training on an encoder f ( · ) to encode each window into a latent representation z i ∈ R N × d , where d indicates the dimension of the latent representation. More specific, PowerPM consists of an exogenous variable enhanced temporal encoder f T ( · ) and a hierarchical encoder f H ( · ) , with the process: z i = f ( x i , o i , G ) = f H ( f T ( x i , o i ) , G ) . In addition, a novel self-supervised strategy which combines masked ETS modeling and dual-view contrastive learning is used for pre-training PowerPM. Next, we will detail the techniques in both model architecture and pre-training strategy.

2.1 Hierarchical Graph Construction

The data of cities, districts, and users in ETS data naturally form a hierarchical relationship, based on which we can construct a hierarchical graph. However, the imbalance in the number of users and districts means there will be multitude of edges between user nodes and district nodes, which significantly increases the complexity of graph modeling. To address this, we employ a clustering strategy to create intermediary nodes, which is a common approach to implement graph sparsification [13] and a user group policy in the power systems [36, 44, 12]. As depicted in Fig. 3 (c), we use clustering method to categorize users into several clusters, the detailed process can be found in App. B.1. The cities are bidirectionally connected to districts, and these user clusters are also bidirectionally connected to districts but are unidirectionally connected to districts. By sparsifying the edges, we enhance the efficiency of graph modeling. Mathematically, we represent the hierarchy as a directed graph G = ( V , E , R ) , where V is the set of nodes, each node corresponds to an instance, E is the set of directed edges, and R is the set of type of edges (e.g. user cluster → district, district → user, etc.).

2.2 Temporal Encoder with Exogenous Variables

Patching. In the G , each node's feature x i is a window of ETS data corresponding to instance i . Due to the semantic sparsity of time series, we patch each window x i into N p segments, each of length P , resulting in p i ∈ R N p × P , where N p = ⌈ T w -P S ⌉ +1 , and this method proved its validity in many works [21, 17, 20]. Subsequently, a linear projection is applied to each segment to obtain the window representation h i ∈ R N p × d .

Exogenous Variables Encoding. To efficiently interact with exogenous variables, we model these variables using learnable embeddings E ∈ R ( ∑ K -1 k =0 M k ) × d , where K indicates the number of exogenous variables (e.g. weather type and temperature), M k represents the number of value types of the k -th exogenous variable (e.g. sunny and rainy in weather type variable). The exogenous variables o ( k ) i ∈ R N p × P corresponding to p i of the k -th exogenous variable are used to obtain representations of the exogenous variables from E , indexing out e ( k ) i ∈ R N p × d , as illustrated in

Fig. 3 (b). Subsequently, we derive a representation u i ∈ R N p × d that considers the window's exogenous variable influence: u i = h i + ∑ K -1 k =0 e ( k ) i .

Temporal Encoder. To model the complex temporal dependency and interaction with exogenous variables, we use the vanilla Transformer encoder [34] to encode u i , resulting in an augmented temporal representation ˆ z i ∈ R N p × d .

2.3 Hierarchical Encoder

To model the complex correlation across different hierarchies, we employ Graph Neural Networks (GNNs). GNNs have recently become increasingly popular for modeling relationships within time series data, which enhances temporal representation [7, 26, 40]. In addition, considering that the correlation relationships of different edges are distinct, we adopt R-GCN [25] to integrate information across various hierarchies and instances, as depicted in Fig 3 (a). Specifically, we use R-GCN to update the representation ˆ z by considering its neighboring nodes in G , with the final node representation denoted as z i ∈ R N p × d . Moreover, we use z i to perform self-supervised pre-training.

2.4 Self-supervised Pre-training

2.4.1 Masked ETS Modeling

To model temporal dependency within an ETS window, we have adopted the widely utilized masked reconstruction strategy. Nevertheless, existing random masking methods may face a significant challenge: they reconstruct the missing part based on the known surrounding part [21, 8], without considering the prediction of future parts relying solely on the past part. This approach not only diminishes the difficulty of the pre-training stage but also lacks consistency across pre-training task and forecasting task.

To address this issue, we propose a novel masking approach that combines random and casual masking, as shown in Fig. 3 (d) ( left ). Specifically, we randomly select one of the masking approaches for a given patched window p i , resulting in masked p i . This approach not only retains the benefits of the random masking strategy but also ensures that the model learns to predict future parts based solely on past information, thereby it can more comprehensively capture the temporal dependencies within a window. Mathematically, this can be formulated as: masked p i = { Mask r ( p i ) if α < 0 . 5 Mask c ( p i ) otherwise ,

where Mask r and Mask c denote the random and causal masking, respectively, and α ∈ [0 , 1] is a uniformly distributed variable. Specifically, after the x i is inputted into PowerPM for masked ETS modeling, we will obtain a reconstructed ˆ x i . The corresponding reconstruction loss is: L MSE = 1 N ∑ N i =1 ( x i -ˆ x i ) 2 .

2.4.2 Dual-view Contrastive Learning

The objective of contrastive learning is to learn representations by bringing positive pairs closer and pushing negative pairs farther apart in the latent space [5, 6]. Motivated by this, to make PowerPM aware of the discrepancy across ETS windows, we employ dual-view contrastive learning (DVCL) to discern subtle differences in electricity usage behavior.

Positive and Negative Sample Pairs. These pairs are determined from two views: one is temporal view , which is based on the time difference between the two windows. Another is the instance view , which depends on whether two windows belong to the same instance. For the same instance, the closer the time difference between two windows, the closer their representations are likely to be. This idea is also presented in [31, 42]. Conversely, windows from different instances or the same instance with a larger time difference are likely to have more distinct representations. Overall, we consider adjacent windows from the same instance as positive samples, while windows from different instances or non-adjacent windows from the same instance are negative samples. As depicted in Fig. 3 (d) ( right ), for the district node V in G , the original start timestamp about this window is T a . After shifting several time steps δ on, we obtain another window V + starting at T a + δ , which serves as a positive sample. Meanwhile, we select windows from other nodes in G , such as city P , starting at T a , as well as windows from the same node V but starting at T c , where | T c -T a |≫ δ . These windows serve as instance and temporal negative samples, respectively, and are denoted as P -and V -.

Mathematically, given an ETS window x i , we obtain a positive sample x + i by shifting it by δ time steps. The other samples in this batch serve as negative samples, totaling B -1 negative samples, where B is the batch size during pre-training. The DVCL loss is: L DVCL = -∑ N i =1 log exp ( sim ( f ( x i ) ,f ( x + i ) ) /τ ) ∑ B m =1 I · exp( sim ( f ( x i ) ,f ( x m )) /τ ) , where I is the boolean vector to select the negative pairs and sim ( · ) is cosine similarity function.

3 Experiments

3.1 Experiment Setup

Pre-training Dataset. PowerPM is pre-trained on a mount of ETS data, a private dataset from the real scenario 1 . This pre-training dataset encompasses ETS data of cities, districts, and users, covering over 3 years records. The ETS data is collected at a frequency of one data point every 15 minutes. More details are in App. A

Downstream Dataset. To evaluate the performance of PowerPM, we conduct comprehensive experiments on eleven downstream private and public datasets. And seven private datasets are also collected from real scenario. These datasets have different labels for different tasks. Among them, the solar generation dataset does not have a hierarchical structure due to its particularity. Four public datasets are obtained from CSISO 2 , ISONE 3 , NYISO 4 , and PJM 5 , and they all exhibit a hierarchical structure. Further details can be found in Appendix A.

Settings. For the model configurations, the temporal encoder contains a 26 -layer Transformer encoder with model dimension 1024 , inner dimension (FFN) 2048 and 16 attention heads, and the hierarchical encoder contains 2 -layer R-GCN. PowerPM contains about 250M parameters. During pre-training, the 40% segments in each input window are masked in the form of random mask and casual mask, the user cluster numbers is set to 12 . See further details in App. B.1

Baselines. We compare with 8 state-of-the-art methods: Large Language Model (LLM) enhanced models: GPT4TS [51], Time-LLM [17], UniTime [20]; pre-train models: PatchTST [21], CoST [37], TS2Vec [42]; supervised models: DLinear [43], TimesNet [38]. More implementation details are provided in App. B.2.

Evaluation Metrics. For forecasting and imputation tasks, we use mean squared error (MSE): 1 n ∑ n i =1 ( y -ˆ y ) 2 and mean absolute error (MAE): 1 n ∑ n i =1 | y -ˆ y | as the evaluation metric. For classification tasks, we use accuracy as the metric. The metric of the anomaly detection task includes precision, recall, F 0 . 5 , and F 1 scores. The Fβ is a metric defined as the weighted harmonic mean of precision and recall, with the following equation: Fβ = ( 1+ β 2 ) × precision × recall β 2 × precision + recall . We use F 0 . 5 for anomaly detection, since precision is more important than recall in power systems scenario [15].

3.2 Downstream Tasks

Demand-side Management. Demand-side management aims to optimize and balance the power system by managing and adjusting the electricity demand of end-users. We develop tasks to predict load at different levels (such as cities and users) and tasks to forecast solar generation. With demandside management, we can better plan and schedule power resources, improve energy efficiency, promote the development of renewable energy, and achieve sustainable energy management.

Grid Stability. To ensure the stability of the power grid, we have implemented a series of tasks, including electricity theft detection, load imputation, and clock anomaly detection, to address the impact of potential appliance failures within the grid and external electricity theft on the quality of power data and grid operations. Internal appliance malfunctions within the grid such as clock anomalies or the inability to record electricity usage accurately decrease the accuracy of power data, making it challenging for power dispatch and management. Additionally, external electricity theft

can cause economic losses and pose a threat to the stable operation and reliability of the power grid, potentially causing power outages and other adverse effects.

Consumer Behavior Analysis. To provide users with more assistance, we have implemented tasks such as detection of elderly living alone, high-power appliance detection, gender classification, age classification, and family structure classification. Additionally, we can provide more flexible power scheduling plans for special groups, so as to optimize power dispatch. We also aim to understand the energy usage differences among different genders and age groups and provide personalized energy management recommendations and services for different users.

.

Tasks

PowerPM

PowerPM

freeze

GPT4TS [51]

TimeLLM [17]

UniTime [20]

PatchTST [21]

CoST [37]

TS2Vec [42]

TimesNet [38]

DLinear [43]

MSE

MSE

MSE

MSE

MSE

MSE

MSE

MSE

MSE

MSE

Demand-side Management

Exclusive User

Forecasting

4

0.3378

0.3557

0.4102

*0.3923

0.4165

0.3929

0.4197

0.4891

0.4335

0.4228

96

0.4183

0.4354

0.4682

0.4832

*0.4514

0.4600

0.5166

0.5453

0.5123

0.5398

288

0.4770

0.5026

0.5319

0.5207

0.5370

*0.5173

0.5634

0.5679

0.5569

0.5818

672

0.5476

0.5831

0.5840

*0.5789

0.5899

0.5347

0.6088

0.6013

0.5961

0.6301

0.4452

0.4692

0.4986

0.4938

0.4987

*0.4762

0.5271

0.5509

0.5247

0.5436

Public User

Forecasting

4

0.2353

0.2507

0.3044

*0.2857

0.2967

0.2911

0.4076

0.3598

0.3583

0.3592

96

0.2604

*0.3142

0.3456

0.3021

0.3645

0.3211

0.4395

0.4054

0.3974

0.4567

288

0.3226

*0.3478

0.3914

0.3449

0.4050

0.3735

0.5128

0.5276

0.4359

0.5455

672

0.3818

*0.4061

0.4470

0.3720

0.4424

0.4325

0.5565

0.5756

0.5271

0.5960

0.3000

*0.3297

0.3721

0.3262

0.3772

0.3546

0.4791

0.4671

0.4297

0.4894

District

Forecasting

4

0.2382

0.2736

0.3239

*0.2924

0.3115

0.3489

0.3837

0.3989

0.4135

0.3701

96

0.2926

0.3348

0.3521

*0.3434

0.3532

0.3891

0.4166

0.4507

0.4742

0.4413

288

0.3300

*0.3760

0.3836

0.3656

0.3903

0.4458

0.4455

0.4836

0.4950

0.5186

672

0.3710

0.4199

*0.4110

0.3940

0.4213

0.4852

0.5109

0.5402

0.5513

0.6004

0.3080

*0.3511

0.3677

0.3489

0.3691

0.4173

0.4392

0.4684

0.4835

0.4826

City

Forecasting

4

0.1725

0.2213

0.2754

0.2620

*0.2435

0.2654

0.2757

0.2650

0.2455

0.3442

96

0.2272

0.2818

0.2958

0.2885

0.2910

*0.2858

0.3065

0.2894

0.3030

0.4084

288

0.2484

0.3371

0.3311

0.3390

*0.3365

0.3682

0.3540

0.3468

0.3976

0.4471

672

0.3211

0.3706

0.3746

0.3933

*0.3727

0.4256

0.4313

0.4646

0.4622

0.5196

0.2423

0.3027

0.3192

0.3207

*0.3109

0.3363

0.3419

0.3415

0.3521

0.4298

Solar Generation

Forecasting

4

0.0993

0.1131

0.1219

0.1315

0.1561

*0.1188

0.1678

0.2330

0.3379

0.4177

96

0.1223

0.1646

0.1894

0.2183

0.2468

*0.1766

0.3822

0.3394

0.4216

0.4710

288

0.2337

0.2679

0.2330

0.2862

0.3366

*0.2538

0.4568

0.3958

0.4570

0.5472

672

0.3076

*0.3438

0.2893

0.3561

0.3843

0.3607

0.4984

0.4259

0.5128

0.5993

0.1907

*0.2224

0.2084

0.2480

0.2810

0.2275

0.3763

0.3485

0.4323

0.5088

Grid Stability

Exclusive User

Imputation

0.125

0.2459

0.2832

0.2902

0.2442

*0.2673

0.2820

0.3243

0.3636

0.3334

0.3702

0.25

0.2621

*0.3136

0.3448

0.3036

0.3398

0.3318

0.3615

0.4150

0.3882

0.4139

0.375

0.3288

0.3573

0.4025

0.3754

0.4080

*0.3725

0.4105

0.4595

0.4275

0.4634

0.5

0.3661

0.4125

0.4342

0.4243

0.4393

*0.4190

0.4805

0.5036

0.5103

0.5365

0.3007

*0.3417

0.3679

0.3369

0.3636

0.3513

0.3942

0.4354

0.4149

0.4460

Public User

Imputation

0.125

0.2348

*0.2651

0.2897

0.2614

0.2987

0.3070

0.3516

0.3223

0.3006

0.3544

0.25

0.2776

*0.2949

0.3327

0.2837

0.3340

0.3667

0.4011

0.3888

0.3583

0.4013

0.375

0.3237

*0.3320

0.4005

0.3044

0.3505

0.4105

0.4420

0.4316

0.4136

0.4487

0.5

0.3919

*0.4295

0.4623

0.3776

0.4439

0.4423

0.4846

0.5028

0.5235

0.5497

0.3070

*0.3304

0.3713

0.3068

0.3568

0.3816

0.4198

0.4114

0.3990

0.4385

District

Imputation

0.125

0.0811

0.1212

*0.1225

0.1364

0.1653

0.1506

0.1852

0.2222

0.1766

0.2332

0.25

0.1284

0.1689

0.2016

*0.1710

0.2698

0.2679

0.2881

0.3042

0.2669

0.2810

0.375

0.1666

0.2223

0.2430

*0.2381

0.3132

0.3272

0.3432

0.3524

0.3598

0.3409

0.5

0.2269

0.2938

0.3238

*0.3068

0.3591

0.3938

0.4249

0.4227

0.4053

0.4051

0.1508

0.2016

0.2227

*0.2131

0.2769

0.2849

0.3104

0.3254

0.3022

0.3151

City

Imputation

0.125

0.0753

*0.1250

0.1101

0.1465

0.1502

0.1807

0.2161

0.2476

0.1825

0.2542

0.25

0.1114

*0.1626

0.1524

0.1912

0.2047

0.2313

0.2715

0.2885

0.2237

0.2987

0.375

0.1451

0.2155

*0.2175

0.2409

0.2557

0.2714

0.3262

0.3313

0.2740

0.3663

0.5

0.2412

*0.2623

0.2357

0.2965

0.3034

0.3417

0.3728

0.3935

0.3389

0.4134

0.1433

*0.1914

0.1789

0.2188

0.2285

0.2563

0.2967

0.3152

0.2548

0.3332

Electricity Theft

Detection

Pre.

0.3793

0.3213

0.2865

0.2537

0.2515

0.2678

*0.3149

0.3076

0.2790

0.2603

Rec.

0.5911

0.5487

0.4444

0.4991

0.5009

0.4665

*0.5281

0.4943

0.4448

0.4594

F0.5

0.4086

0.3503

0.3084

0.2814

0.2793

0.2927

*0.3426

0.3327

0.3015

0.2850

F1

0.4621

0.4053

0.3484

0.3364

0.3349

0.3403

*0.3945

0.3792

0.3429

0.3323

Clock Anomaly

Detection

Pre.

0.4540

0.3874

0.3247

0.3108

0.3294

0.2321

0.3620

*0.3859

0.2341

0.1719

Rec.

0.7881

0.7391

0.7255

0.7120

0.6908

0.6290

0.7309

*0.7326

0.5571

0.5432

F0.5

0.4961

0.4281

0.3650

0.3503

0.3679

0.2656

0.4026

*0.4262

0.2648

0.1991

F1

0.5761

0.5083

0.4486

0.4327

0.4461

0.3391

0.4842

*0.5055

0.3297

0.2612

Consumer Behavior Analysis

High Power

Appliance Detection

Pre.

0.7427

*0.7265

0.6951

0.6988

0.7430

0.6538

0.6973

0.6880

0.7027

0.6008

Rec.

0.5832

*0.5426

0.4924

0.5024

0.5375

0.4773

0.5715

0.5116

0.5292

0.4668

F0.5

0.7042

*0.6804

0.6422

0.6481

0.6902

0.6088

0.6679

0.6436

0.6595

0.5682

F1

0.6534

0.6212

0.5765

0.5845

*0.6238

0.5518

0.6282

0.5868

0.6037

0.5254

Elderly Alone

Detection

Pre.

0.4540

*0.4374

0.4677

0.4135

0.4254

0.3301

0.3826

0.3588

0.3025

0.2282

Rec.

0.7881

0.7587

*0.7355

0.6898

0.7044

0.6448

0.6796

0.6690

0.6934

0.5704

F0.5

0.4961

*0.4779

0.5044

0.4495

0.4620

0.3658

0.4192

0.3955

0.3409

0.2593

F1

0.5761

*0.5549

0.5718

0.5171

0.5305

0.4367

0.4896

0.4671

0.4212

0.3260

Gender CLS

Acc.

0.7571

0.7142

*0.6466

0.6340

0.6328

0.5490

0.6402

0.5960

0.5079

0.4786

Age CLS

Acc.

0.6830

0.6418

0.6295

0.6001

0.5774

0.5134

*0.6298

0.5864

0.5379

0.5187

Family Structure CLS

Acc.

0.6406

*0.6129

0.5974

0.5687

0.6179

0.5205

0.6062

0.5463

0.5038

0.4840

3.3 Main Results

Overview. As a foundation model for power systems, PowerPM achieves SOTA performance on various tasks when compared to other baseline models, highlighting its ability to generalize effectively across a wide range of tasks. We derive more detailed comparisons of each task in the following paragraphs, and in all tables we mark the best results in bold , the second-best in underlined, and the third-best in ∗ asterisk in each column.

Demand-side Management. The forecasting results for load and solar generation are presented in Tab. 1 (upper part). The results cover various forecast horizons, including 4 ( 1 hour), 96 ( 1 day), 288

( 3 days), and 672 ( 1 week). The choice of these forecast horizons holds physical significance as it aligns with real-world scenarios. The results demonstrate that not only PowerPM achieves near SOTA performance, but also PowerPM freeze surpasses most baseline models. This highlights the superiority of PowerPM in modeling temporal dependencies and capturing the impact of exogenous variables through the use of a temporal encoder and a novel masked ETS modeling approach. Furthermore, PowerPM attains near SOTA performance at different hierarchical levels, particularly at the macro level (district and city), highlighting the importance of modeling the hierarchical correlation within ETS data in PowerPM. Notably, among the baselines, none of the baselines capture the hierarchical correlation of ETS data, resulting in a performance decrease in comparison to PowerPM.

Grid Stability. To assess the efficacy of PowerPM in grid stability application, we conduct comprehensive experiments encompassing load imputation across various masked ratios ( 12 . 5% , 25% , 37 . 5% , 50% ), anomaly detection (including electricity theft and clock anomaly detection), encompassing a total of 18 tasks. The results, detailed in Tab. 1 (middle part), illustrate PowerPM's consistent superiority over all baselines, with the PowerPM freeze variant also surpassing the majority of baselines. Notably, in imputation tasks, PowerPM demonstrates marked superiority over other pre-trained models (such as PatchTST and CoST), underscoring the advantages of hierarchical modeling in ETS data. Furthermore, in anomaly detection tasks, as shown in Tab. 1 (middle part), our model consistently achieves near-optimal results. While GPT4TS records the highest F0.5 score among the baseline methods, attributed to its generation of GPT-2, PowerPM further enhances the F0.5 score over GPT4TS. This improvement stems from our temporal encoder's broader receptive field and the hierarchical encoder's capacity to capture hierarchical correlations across all levels, which are both pivotal for modeling ETS data.

Consumer Behavior Analysis. We explore two anomaly detection tasks: elderly living alone and high-power appliance detection, and three classification tasks: gender, age, and family structure classification. The results in Tab. 1 (bottom part) demonstrate PowerPM's SOTA performance, illustrating its capacity for deep semantic insight and contextual awareness. Furthermore, PowerPM freeze sustains high performance, highlighting the model's innate ability to extract and generalize features.

3.4 Model Analysis

Generalization Ability Analysis. To further verify the generalization ability of PowerPM on more datasets from other domains, we evaluate PowerPM on 4 public datasets mentioned above. The results in Tab. 2 demonstrate that PowerPM outperforms nearly all SOTA methods and PowerPM freeze surpasses most SOTA methods, highlighting the generalization superiority of PowerPM.

Ablation Study. To assess the effectiveness of each component in our model, we conduct several ablation experiments. Specifically, we remove the following components from our model to examine their effects on performance: the hierarchical encoder (PowerPM-H), the dual-view contrastive learning strategy (PowerPMC), and the exogenous variables encoding module (PowerPM-E). Furthermore, we replace the masked ETS modeling module with vanilla random masking (PowerPM-M). We categorize 44 tasks into four traditional time series tasks: forecasting, missing value imputation, anomaly detection, and classification. The evaluation metrics are Mean Squared Error (MSE) for forecasting and missing value imputation, F0.5 score is for anomaly detection, and accuracy (Acc.) for classification. The performance is averaged to provide a comprehensive assessment.

The results of the ablation study are in Fig. 4 (a). The results indicate that PowerPM outperforms its vari-

0.526

0.504

0.487

0.461

0.487

0.54

0.50

0.46

0.42

0.36

0.32

0.28

0.324

0.360

0.359

0.345

0.349

0.225

0.272

0.263

0.252

0.262

0.28

0.24

0.20

Full

-H

-M

-C

-E

0.694

0.670

0.655

0.661

0.641

0.70

0.66

0.62

0.58

60

30

10

0.65

0.55

0.45

0.35

0.25

0.45

0.40

0.35

0.30

0.25

UniTime

PatchTST

TimeLLM

PowerPM

0.55

0.45

0.35

0.70

0.60

0.50

0.40

30

70

120

250

0.70

0.65

0.60

0.55

0.50

0.45

0.40

0.35

0.30

0.475

0.425

0.375

0.325

0.275

0.40

0.35

0.30

0.25

Forecasting

Imputation

Anomaly Detection

Classification

MSE

MSE

MSE

MAE

MSE

MSE

MSE

MAE

F0.5

F0.5

F0.5

F1

Rec.

Pre.

Acc.

Acc.

Acc.

Data Proportion (%)

Ablation Study

(a)

(b)

(c)

Few-shot Learning

ants, providing evidence for the contribution of each component. Among them, PowerPM-H exhibits the most substantial decrease in performance compared to the full PowerPM, emphasizing the significance of interactions between micro- and macro-levels when modeling hierarchical ETS data. The observed performance degradation of PowerPM-M, particularly in forecasting tasks, shows that causal masking can capture more complex temporal dependency. Moreover, the declined performance of PowerPM-C, particularly in anomaly detection and classification tasks, suggests that dual-view contrastive learning is effective in capturing subtle discrepancies between instances. Furthermore, PowerPM-E also presents performance degradation. This emphasizes the effectiveness of the exogenous variables encoding module in capturing the impact of exogenous factors. For detailed results of 44 tasks, please refer to App. 7.

Few-shot Learning. In power systems, collecting abundant ETS data for downstream tasks is a significant investment. To demonstrate the practical application value of our work, we conduct a performance comparison between PowerPM and baseline models, considering the limited availability of ETS data. Specifically, models are fine-tuned on 10% , 30% and 60% of the downstream dataset, respectively. Similar to an ablation study, we group our results by task type, which can be seen in Fig. 4 (b). The performance of PowerPM exhibits a slight decrease when there is a significant reduction in the proportion of fine-tuning data. This observation serves as evidence of the effectiveness of our novel pre-training strategy. Additionally, it highlights that the PowerPM adeptly captures temporal dependencies and hierarchical correlations present in the ETS data during pre-training, enabling easier adaptation to downstream tasks. More detailed results can be referred in App. 8.

Model Scale Evaluation. To explore the impact of model size on performance, we design three variants of PowerPM (about 250 M ) with smaller sizes: PowerPM-Tiny (about 30 M ), PowerPMSmall (about 70 M ), PowerPM-Medium (about 120 M ), and pre-train them on the same datasets. For the pre-training details, please refer to App. B.1. After pre-training, we evaluate these variants on all downstream tasks and present the results by task type like the ablation study. As shown in Fig. 4 (c), as the size of the model increases, we observe an overall improvement of the performance on all downstream tasks. Specifically, PowerPM outperforms the other variants in all metrics. In addition, larger models exhibit almost a decrease in standard deviation, indicating a more stable performance.

Therefore, the utilization of a larger model with higher capacity and large ETS data enables better generalization across a wide range of downstream tasks.

4 Related Work

Self-supervised Pre-training Model. Large-scale model based on self-supervised pre-training has become more significant in both industrial and academic domains due to the versatility and impressive performance. It initially developed in the fields of computer vision [14] and natural language processing [8, 11]. Self-supervised pre-training in time series is typically classified into two paradigms: contrastive learning and mask modeling. The objective of contrastive learning is to learn representation by pushing positive pairs closer and pull negative pairs away in the embedding space [16]. TS2Vec [42] proposes contextual consistency for positive pair selection. Then, CoST [37] extracts the trend and seasonal feature representations, and takes advantage of both time and frequency domain contrastive loss to encourage discriminative seasonal representation. And TF-C [47] applies time-frequency consistency for embedding time-based and frequency-based neighbors. In mask modeling, to extract the contextual semantic information, PatchTST [21] masks at the series-level.

Supervised Learning Model. Since the self-attention mechanism in Transformer [33] showed the great ability to seize global dependencies between input and output, recently many variants have been proposed to tackle power system tasks. LogTrans [19], Informer [48] reduce the complexity by optimizing the vanilla self-attention mechanism. Autoformer [39] leverages auto-correlation mechanism to achieve series-wise representation aggregation. FEDformer [50] incorporates frequency-domain information to enhances prediction performance while reducing complexity to linear levels. Besides, DLinear [43] questions the effectiveness of transformers as it outperforms most Transformer-based SOTAs, with a simple linear model. TimesNet [38] has treated time series as a 2 D signal and utilized a convolution-based inception net backbone to function as a comprehensive time series model.

Large Language Models Enhanced Model. Recently, the advancement of Large Language Models (LLMs) has opened up new horizons in time series modeling. Many LLMs, such as llama [32], GPT3 [11], GPT-4 [1], ChatGLM [9] have the capability to capture complex dependencies and understand varied textual data, yielding sensible reasonable generation results. Therefore, many reserachers begin to apply LLMs to assist time series modeling. Time-LLM [17] and TEXT [28] employ reprogrammed input time series with text prototype embedding and incorporate textual prompts for time series. GPT4TS [51] and UniTime [20] apply fine-tuning to selected components of LLMs to improve performance in time series analysis tasks. TEMPO [3] incorporates the decomposition of time series and retrieval-based prompt design for non-stationary time series data.

However, despite numerous methods for self-supervised and supervised time series, the research on foundation models specifically designed for power systems remains relatively sparse. And LLMs are limited in power systems scenario, lacking enough textual descriptions for domain knowledge.

5 Conclusion

This paper introduces the PowerPM, a foundational model designed to model ETS data within power systems. PowerPM consists of a temporal encoder and a hierarchical encoder . Furthermore, PowerPM leverages a novel self-supervised pre-training framework consisting of masked ETS modeling and dual-view contrastive learning . Our experiments involve two real-world scenario datasets, comprising private and public data. Through pre-training on massive ETS data, PowerPM achieves SOTA performance on diverse downstream tasks within the private dataset. Moreover, when transferred to the public dataset, PowerPM maintains its superiority, showcasing its remarkable generalization ability across various tasks and domains. Further analysis shows the effectiveness of a foundation model in the field of power system. Also, PowerPM is an off-the-shelf model with its code and weights. This feature greatly mitigates the challenges associated with sample and label efficiency, allowing it to be directly integrated into various power system applications.

Acknowledgments

This work was partially supported by National Natural Science Foundation of China (No. 62322606, No. 62441605).

References

Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, Ilge Akkaya, Florencia Leoni Aleman, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anadkat, et al. Gpt-4 technical report. arXiv preprint arXiv:2303.08774 , 2023.

Vadim Arzamasov, Klemens Böhm, and Patrick Jochem. Towards concise models of grid stability. In 2018 IEEE international conference on communications, control, and computing technologies for smart grids (SmartGridComm) , pages 1-6. IEEE, 2018.

Defu Cao, Furong Jia, Sercan O Arik, Tomas Pfister, Yixiang Zheng, Wen Ye, and Yan Liu. Tempo: Prompt-based generative pre-trained transformer for time series forecasting. arXiv preprint arXiv:2310.04948 , 2023.

Widyaning Chandramitasari, Bobby Kurniawan, and Shigeru Fujimura. Building deep neural network model for short term electricity consumption forecasting. In 2018 International Symposium on Advanced Intelligent Informatics (SAIN) , pages 43-48. IEEE, 2018.

Ting Chen, Simon Kornblith, Mohammad Norouzi, and Geoffrey Hinton. A simple framework for contrastive learning of visual representations. In ICML , 2020.

Xinlei Chen and Kaiming He. Exploring simple siamese representation learning. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition , pages 15750-15758, 2021.

Yue Cui, Kai Zheng, Dingshan Cui, Jiandong Xie, Liwei Deng, Feiteng Huang, and Xiaofang Zhou. Metro: A generic graph neural network framework for multivariate time series forecasting. Proc. VLDB Endow. , 2021.

Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding. NAACL , 2018.

Zhengxiao Du, Yujie Qian, Xiao Liu, Ming Ding, Jiezhong Qiu, Zhilin Yang, and Jie Tang. Glm: General language model pretraining with autoregressive blank infilling. arXiv preprint arXiv:2103.10360 , 2021.

Xi Fang, Satyajayant Misra, Guoliang Xue, and Dejun Yang. Smart grid-the new and improved power grid: A survey. IEEE communications surveys & tutorials , 14(4):944-980, 2011.

Tianyu Gao, Adam Fisch, and Danqi Chen. Making pre-trained language models better few-shot learners. IJCNLP , 2020.

Benjamin Goehry, Yannig Goude, Pascal Massart, and Jean-Michel Poggi. Aggregation of multi-scale experts for bottom-up load forecasting. IEEE Transactions on Smart Grid , 2020.

Mohammad Hashemi, Shengbo Gong, Juntong Ni, Wenqi Fan, B. Aditya Prakash, and Wei Jin. A comprehensive survey on graph reduction: Sparsification, coarsening, and condensation, 2024.

Kaiming He, Xinlei Chen, Saining Xie, Yanghao Li, Piotr Dollár, and Ross Girshick. Masked autoencoders are scalable vision learners. CVPR , 2022.

Wenjie Hu, Yang Yang, Jianbo Wang, Xuanwen Huang, and Ziqiang Cheng. Understanding electricity-theft behavior via multi-source data. In Proceedings of The Web Conference 2020 , pages 2264-2274, 2020.

Ashish Jaiswal, Ashwin Ramesh Babu, Mohammad Zaki Zadeh, Debapriya Banerjee, and Fillia Makedon. A survey on contrastive self-supervised learning. Technologies , 9(1):2, 2020.

Ming Jin, Shiyu Wang, Lintao Ma, Zhixuan Chu, James Y Zhang, Xiaoming Shi, Pin-Yu Chen, Yuxuan Liang, Yuan-Fang Li, Shirui Pan, et al. Time-llm: Time series forecasting by reprogramming large language models. arXiv preprint arXiv:2310.01728 , 2023.

Diederik P. Kingma and Jimmy Ba. Adam: A method for stochastic optimization. In ICLR (Poster) , 2015.

Shiyang Li, Xiaoyong Jin, Yao Xuan, Xiyou Zhou, Wenhu Chen, Yu-Xiang Wang, and Xifeng Yan. Enhancing the locality and breaking the memory bottleneck of transformer on time series forecasting. Advances in neural information processing systems , 32, 2019.

Xu Liu, Junfeng Hu, Yuan Li, Shizhe Diao, Yuxuan Liang, Bryan Hooi, and Roger Zimmermann. Unitime: A language-empowered unified model for cross-domain time series forecasting. In Proceedings of the ACM Web Conference 2024 , 2024.

Yuqi Nie, Nam H Nguyen, Phanwadee Sinthong, and Jayant Kalagnanam. A time series is worth 64 words: Long-term forecasting with transformers. ICLR , 2023.

Peter Palensky and Dietmar Dietrich. Demand side management: Demand response, intelligent energy systems, and smart loads. IEEE transactions on industrial informatics , 7(3):381-388, 2011.

Yue Pang, Bo Yao, Xiangdong Zhou, Yong Zhang, Yiming Xu, and Zijing Tan. Hierarchical electricity time series forecasting for integrating consumption patterns analysis and aggregation consistency. In IJCAI , pages 3506-3512, 2018.

Adam Paszke, S. Gross, Francisco Massa, A. Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Z. Lin, N. Gimelshein, L. Antiga, Alban Desmaison, Andreas Köpf, Edward Yang, Zach DeVito, Martin Raison, Alykhan Tejani, Sasank Chilamkurthy, Benoit Steiner, Lu Fang, Junjie Bai, and Soumith Chintala. Pytorch: An imperative style, high-performance deep learning library. In NeurIPS , 2019.

Michael Schlichtkrull, Thomas N Kipf, Peter Bloem, Rianne Van Den Berg, Ivan Titov, and Max Welling. Modeling relational data with graph convolutional networks. In The Semantic Web: 15th International Conference, ESWC 2018, Heraklion, Crete, Greece, June 3-7, 2018, Proceedings 15 , pages 593-607. Springer, 2018.

Chao Shang, Jie Chen, and Jinbo Bi. Discrete graph structure learning for forecasting multiple time series. In International Conference on Learning Representations , 2021.

Arunesh Kumar Singh, S Khatoon, Md Muazzam, DK Chaturvedi, et al. Load forecasting techniques and methodologies: A review. In 2012 2nd International Conference on Power, Control and Embedded Systems , pages 1-10. IEEE, 2012.

Chenxi Sun, Yaliang Li, Hongyan Li, and Shenda Hong. Test: Text prototype aligned embedding to activate llm's ability for time series. arXiv preprint arXiv:2308.08241 , 2023.

Xiaorong Sun, Peter B. Luh, Kwok W. Cheung, Wei Guan, Laurent D. Michel, S. S. Venkata, and Melanie T. Miller. An efficient approach to short-term load forecasting at the distribution level. IEEE Transactions on Power Systems , 2016.

Yuechuan Tao, Jing Qiu, Shuying Lai, Junhua Zhao, and Yusheng Xue. Carbon-oriented electricity network planning and transformation. IEEE Transactions on Power Systems , 36(2):10341048, 2020.

Sana Tonekaboni, Danny Eytan, and Anna Goldenberg. Unsupervised representation learning for time series with temporal neighborhood coding. In International Conference on Learning Representations , 2021.

Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timothée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, et al. Llama: Open and efficient foundation language models. arXiv preprint arXiv:2302.13971 , 2023.

Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Łukasz Kaiser, and Illia Polosukhin. Attention is all you need. Advances in neural information processing systems , 30, 2017.

Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Ł ukasz Kaiser, and Illia Polosukhin. Attention is All you Need. In Advances in Neural Information Processing Systems , 2017.

Hong Wang, Khalid A. Alattas, Ardashir Mohammadzadeh, Mohammad Hosein Sabzalian, Ayman A. Aly, and Amir Mosavi. Comprehensive review of load forecasting with emphasis on intelligent computing approaches. Energy Reports , 8, 2022.

Yi Wang, Qixin Chen, Mingyang Sun, Chongqing Kang, and Qing Xia. An ensemble forecasting method for the aggregated load with subprofiles. IEEE Transactions on Smart Grid , 2018.

Gerald Woo, Chenghao Liu, Doyen Sahoo, Akshat Kumar, and Steven Hoi. Cost: Contrastive learning of disentangled seasonal-trend representations for time series forecasting. ICLR , 2022.

Haixu Wu, Tengge Hu, Yong Liu, Hang Zhou, Jianmin Wang, and Mingsheng Long. Timesnet: Temporal 2d-variation modeling for general time series analysis. In The Eleventh International Conference on Learning Representations , 2022.

Haixu Wu, Jiehui Xu, Jianmin Wang, and Mingsheng Long. Autoformer: Decomposition transformers with auto-correlation for long-term series forecasting. NeurIPS , 2021.

Zonghan Wu, Shirui Pan, Guodong Long, Jing Jiang, Xiaojun Chang, and Chengqi Zhang. Connecting the dots: Multivariate time series forecasting with graph neural networks. In Proceedings of the 26th ACM SIGKDD international conference on knowledge discovery & data mining , pages 753-763, 2020.

Dazhi Yang, Gary SW Goh, Siwei Jiang, Allan N Zhang, and Orkan Akcan. Forecast upc-level fmcg demand, part ii: Hierarchical reconciliation. In 2015 ieee international conference on big data (big data) , pages 2113-2121. IEEE, 2015.

Zhihan Yue, Yujing Wang, Juanyong Duan, Tianmeng Yang, Congrui Huang, Yunhai Tong, and Bixiong Xu. TS2Vec: Towards Universal Representation of Time Series. AAAI , 2022.

Ailing Zeng, Muxi Chen, Lei Zhang, and Qiang Xu. Are transformers effective for time series forecasting? In Proceedings of the AAAI conference on artificial intelligence , volume 37, pages 11121-11128, 2023.

Chi Zhang and Ran Li. A novel closed-loop clustering algorithm for hierarchical load forecasting. IEEE Transactions on Smart Grid , 2021.

Hao Zhang, Fan Zhang, Yu Zhang, Hui Cheng, Ruotian Gao, Zongpeng Li, Jiakui Zhao, and Mingzhu Zhang. An elderly living-alone guardianship model based on wavelet transform. In 2022 4th International Conference on Power and Energy Technology (ICPET) , pages 1249-1253. IEEE, 2022.

Huaying Zhang, Qing Wang, Yan Li, Jingwen Ai, Xunyong Hu, Wenhai Zhang, and Dehai Zhang. Clock anomaly detection method of power quality monitoring device based on voltage sag. In 2021 IEEE 2nd China International Youth Conference on Electrical Engineering (CIYCEE) , pages 1-6. IEEE, 2021.

Xiang Zhang, Ziyuan Zhao, Theodoros Tsiligkaridis, and Marinka Zitnik. Self-supervised contrastive pre-training for time series via time-frequency consistency. NeurIPS , 2022.

Haoyi Zhou, Shanghang Zhang, Jieqi Peng, Shuai Zhang, Jianxin Li, Hui Xiong, and Wancai Zhang. Informer: Beyond efficient transformer for long sequence time-series forecasting. In Proceedings of the AAAI conference on artificial intelligence , volume 35, pages 11106-11115, 2021.

Kaile Zhou and Shanlin Yang. Understanding household energy consumption behavior: The contribution of energy big data analytics. Renewable and Sustainable Energy Reviews , 56:810819, 2016.

Tian Zhou, Ziqing Ma, Qingsong Wen, Xue Wang, Liang Sun, and Rong Jin. Fedformer: Frequency enhanced decomposed transformer for long-term series forecasting. In ICML , 2022.

Tian Zhou, Peisong Niu, Liang Sun, Rong Jin, et al. One fits all: Power general time series analysis by pretrained lm. Advances in neural information processing systems , 36, 2024.

A Dataset Description

We conduct experiments on 5 real-world hierarchical electricity time series datasets, one of which was collected from the real scenario. The other four are collected from CSISO 6 , ISONE 7 , NYISO 8 , and PJM 9 . Our experiments include four typical time series analysis tasks on these datasets to evaluate the effect of our approach in both in-domain and cross-domain settings: prediction, missing value imputation, anomaly detection, and classification, which include different sampling frequencies ( 5 minutes, 15 minutes, 1 hour, 1 day). Moreover, it covers a variety of application scenarios in power systems (load forecasting, solar generation forecasting, electricity theft detection and consumer analysis, etc.). Tab. 3 and Tab. 4 summarize the detailed descriptions of these datasets.

A.1 Private Dataset

Private dataset is collected from the load data in the real scenario, covering the period about 6 years. Following data preprocessing, we extract a subset of the data. In order to effectively support our research objectives, we divide the dataset into 9 distinct sub-datasets. One biggest of these sub-datasets is served as the pre-training dataset, while the remaining 7 sub-datasets are utilized as downstream datasets for downstream tasks. These downstream datasets are partitioned into train, validation, and test sets according to a 6 : 2 : 2 ratio, ensuring that the training set contain data from the earlier time period. Further details are provided below:

Pre-training Dataset. The pre-training dataset is derived from a subset of the private dataset, encompassing the period about 4 years.. It consists of unlabeled data recorded at a frequency of one data point every 15 minutes. The dataset is structured hierarchically, including information at the user, district, and city levels.

Load Forecasting and Missing Value Imputation Dataset. This dataset is extracted from a portion of the private dataset about 1 years. The dataset includes hierarchical information at the user, district, and city levels, with data points recorded every 15 minutes. For the missing value imputation task, the dataset is structured to output 672 data points. As for the forecasting task, there are four different prediction horizons: one hour ( 4 data points), one day ( 96 data points), three days ( 288 data points), and seven days ( 672 data points).

Solar Generation Forecasting Dataset. The dataset is collected from many distributed photovoltaic power stations. The dataset has not a hierarchical structure, and data points are recorded at a frequency of one point every 15 minutes. It includes four different prediction horizons: one hour, one day, three days, and seven days.

Electricity Theft Detection Dataset. This dataset comprises the daily electricity consumption records (in K·Wh) of users in 1 year. For each user, the dataset includes the daily aggregate electricity usage. Within the dataset, certain users (referred to as electricity thieves) engage in unauthorized activities involving the electricity meter in order to reduce costs.

Clock Anomaly Dataset. This dataset comprises millions of clock error series, each representing the time deviation, compared to the standard time, and communication delay of various watt-hour meters on a weekly basis. The dataset covers the period about 8 months.

Elderly Living Alone Dataset. This dataset includes the daily electricity consumption records (in K·Wh) of village users. Additionally, employees conduct extensive on-site investigations specifically targeting these users, from which we obtain labels indicating whether each user is an elderly individual living alone or not.

High-power Appliance Detection Dataset. This dataset consists of the daily electricity consumption records (in K·Wh) of village users. Similar to the previous dataset, on-site investigations are conducted by same method, enabling us to collect labels indicating whether each user possesses high-power appliances.

Consumer Analysis Dataset. This dataset contains the daily electricity consumption records (in K·Wh) of village users. Additionally, employees conducted extensive on-site investigations targeting these users, collecting statistics related to the gender of the gender of user who lives alone, the age of the resident elderly, and family structure. The gender labels of user who lives alone are: male and female, totaling two classes; the age labels for residents are: 60 ∼ 70 years old, 70 ∼ 80 years old, 80 ∼ 90 years old, and over 90 years old, totaling four classes; the family structure labels are: 1 people, 2 ∼ 3 people, 4 ∼ 5 people, and more than 6 people, totaling four classes.

A.2 Public Datasets

Four public datasets as cross-domain datasets are selected to validate the generalization ability of our model. These four datasets are named CSISO, ISONE, NYISO, and PJM, which cover 3 types different hierarchical relationships: state-area, region-state, state-city.

CAISO. It is sampled from California, including 34 areas loads and an aggregated load for the state, recorded every hour from April 25 , 2023 , to April 23 , 2024 . The prediction horizons include half a day ( 12 points), one day ( 24 points), and seven days ( 168 points).

ISONE. It is sampled from New England, consisting of 6 states loads and an aggregated load for the region, recorded every hour from October 1 , 2023 , to April 1 , 2024 . The prediction horizons include half a day ( 12 points), one day ( 24 points), and seven days ( 168 points).

NYISO. It is sampled from California, containing 11 areas loads and an aggregated load for the state, recorded every 5 minutes from March 1 , 2023 , to March 31 , 2024 . The prediction horizons include one hour ( 12 points), half a day ( 144 points), and one day ( 288 points).

PJM. It is sampled from 3 states: Florida, Ohio, Washington, which includes 22 cities loads and there 3 state loads, recorded every hour from March 28 , 2023 , to April 26 , 2024 . The prediction horizons include one hour ( 12 points), half a day ( 144 points), and one day ( 288 points).

A.3 Exogenous Variables

We obtained weather and temperature records for all area levels in both the private and public datasets. The weather information from the private dataset is obtained from the Weather Radar 10 . Additionally, the weather information from the public datasets is obtained from the NSF NCAR Research Data Archive 11 . Both sources cover the same timespan as mentioned above, respectively. These records include the maximum and minimum temperatures (in °C for private dataset and ◦ F for public datasets) for each hour in each city.

B PowerPM and Baseline Implementation Details

B.1 PowerPM Implementation

The pre-training stage of the experiment is implemented in PyTorch [24] and conducted on a Linux system with 2 CPUs (AMD EPYC 9654 96 -Core Processor) and 8 GPUs (NVIDIA Tesla A 800 80 G ) for about 8 days. And the downstream task experiment is repeated five times. We select 512 samples as a batch, and every batch contains about 174 k patches, which we set patch len to 48 , stride to 24 . To speed up the model training, we stop the gradient update of the background nodes in the hierarchical graph. We optimize with Adam [18], updating the model parameters every 4 steps, and the model trains for 1310 k updates in total. A reduce learning rate on plateau scheduler is utilized to adjust learning rate during pre-training. Specifically, we set the basic learning rate as 1 e -6 and the maximum learning rate as 2 e -5 , and the learning rate updates for every 10 k updates. In addition, we trained three additional variants of PowerPM with different parameter counts to meet the needs of different users or situations. Detailed model hyperparameters can be found in Tab. 5.

Full Fine-tuning. In the F-FT (Full Fine-tuning) setup, for different tasks, we introduce different head H on the top of pre-trained encoder f ( . ) , where both the parameters of the encoder f ( . ) and the head H are trainable. For forecasting and imputation tasks, we use a prediction H l head to map prediction points or reconstruction points from z i . In this setup, we fine-tune both the head H and the encoder f ( . ) . We utilize 100 %, 60 %, 30 % and 10 % training data for fine-tuning. we utilize a one-layer fully connected network to implement prediction H l and logistic regression from the Sklearn library to implement the classifier H c . The learning rates are specifically set to 4 e -4 and 3 e -5 for public and private datasets.

Partial Fine-tuning. In the P-FT (Partial Fine-tuning) setup, for different tasks, we also introduce different head H on the top of pre-trained encoder f ( . ) . For forecasting and imputation tasks, we use a prediction H l head to map prediction points or reconstruction points from z i . And for anomaly

detection and classfication tasks, a classifier H c on top of the pre-trained encoder f ( . ) . During the whole finetune process, we keep the parameters of f ( . ) fixed. Only the head is fine-tuned in this setup. we utilize a one-layer fully connected network to implement prediction H l and logistic regression from the Sklearn library to implement the classifier H c . The learning rates are specifically set to 4 e -4 and 3 e -5 for public and private datasets.

B.2 Baselines Implementation

We compare with 8 state-of-the-art methods: including Large Language Model (LLM) enhanced models: GPT4TS [51], Time-LLM [17], UniTime [20]; pre-train models: PatchTST [21], CoST [37], TS2Vec [42]; supervised models: DLinear [43], TimesNet [38]. To make a fair and comprehensive comparison, we reproduce all models with official implementation, and use different output head for different downstream tasks. Due to the large scale of the ETS dataset, we increase the number of training epoch and reduce the learning rate in order to make the parameters of the model fully learned.

GPT4TS [51] combines the LLM with Transformer, which use frozen pre-trained GPT-2 for general time series analysis. To implement GPT4TS, we utilized their open-source code, available at https://github.com/DAMO-DI-ML/NeurIPS2023-One-Fits-All. We use the 6 layers of GPT-2, which is proved to have the optimal performance in original paper and the total size of GPT4TS is about 105.15M, and the trainable parameters are 24 . 04 M (GPT-2 is frozen). We set the number of train epochs to 50 , the learning rate to 0 . 0005 , and the batch size to 256 .

Time-LLM [17] frezees the LLM as the backbone, and align time series to text with patch reprogramming. It also designs Prompt-as-Prefix including dataset context, task instruction and input statistics to enrich the input context to direct the transformation of reprogrammed input. We utilized their open-source code, available at https://github.com/KimMeen/Time-LLM to implement Time-LLM. We set the llama-7b with 32 layers as the backbone, which is the most effective recorded in [17] and the total size of Time-LLM is about 7 . 28 B , and the trainable parameters are 58 . 55 M (llama-7b is frozen). To align the dataset context input to our datasets, we constuct different natural language prompt summarized in App. A for private and public datasets, and we set the number of train epochs to 50 , the learning rate to 0 . 005 , and the batch size to 256 .

UniTime [20] leverages LLM to handle time series forecasting across time series domains, which exhibit significant differences in temporal patterns and distribution. The same as dataset context in Time-LLM, UniTime also designs human-crafted instructions to furnish the model with explicit domain identification information. To implement UniTime, we utilized their open-source code, available at https://github.com/liuxu77/UniTime. We implement the backbone LLM with GPT2-small like original paper, and the total size of UniTime is about 108 . 54 M without freeze any parameters. Weuse the same natural language prompt in Time-LLM as the human-crafted instructions for different datasets, and we set the number of train epochs to 50 , the learning rate to 0 . 0005 , the weight decay to 0 . 0001 , and the batch size to 256 .

TS2Vec [42] performs contextual consistency using overlapping subseries and a hierarchical loss function to capture data consistency at the observation and sample levels. We utilize the open-source code available at https://github.com/zhihanyue/ts2vec. Specifically, we set the number of epochs for pre-training to 100 , the learning rate to 0 . 0005 , and the batch size to 512 . Due to the large scale and complex semantics of the pre-trained ETS data, we adjust the representation dimension to 640 , matching the ETS data characteristics. We adopt the default settings provided by the TS2Vec implementation for other settings during pre-training.

CoST [37] comprises both time domain and frequency domain contrastive losses to learn discriminative trend and seasonal representations. We utilize the open-source code available at https://github.com/salesforce/CoST to implement CoST. Specifically, we set the number of epochs for pre-training to 100, the learning rate to 0 . 0005 , representation dimension to 640 , and the batch size to 256 . We adopt the default settings provided by the CoST implementation for other settings during pre-training.

PatchTST [21] changes the input sequence as a series of patch windows, focus the subseries-level attention to capture local semantic information while minimizing memory consumption. We utilize the open-source code available at https://github.com/yuqinie98/PatchTST. For hyperparameters of PatchTST, We set the patch len to 32 and stride to 16 , the number of epochs for pre-training to 100 ,

the learning rate to 0 . 0005 , and the batch size to 512 . We adopt the default settings provided by the PatchTST implementation for other settings during pre-training.

TimeNet [38] is a CNN based time series model which extends the analysis of temporal variations into the 2 D space. It designs TimesBlock with an inception block to extract complex temporal patterns, leading to multiple time series tasks. To implement TimesNet, we utilized their open-source code, available at https://github.com/thuml/Time-Series-Library. Specifically, we set the number of epochs for training to 50 , the learning rate to 0 . 0005 , and the batch size to 128 . We adopt the default settings provided by the TimesNet implementation for other settings for forecasting, imputation classfication anomaly detection .

DLinear [43] decomposes the time series into a trend sequence and a seasonal sequence, then model these two sequences using two simple MLPs. To implement DLinear, we utilized their open-source code, available at https://github.com/cure-lab/LTSF-Linear. Specifically, we set the number of epochs for training to 50 , the learning rate to 0 . 0005 , and the batch size to 512 . We adopt the default settings provided by the DLinear implementation for other settings.

B.3 Cluster Method

We use K-means algorithm to cluster users. Firstly, we get filter out user ETS by labels, and normalize the time series data, represented as an N × M matrix, to ensure that differences in scale do not affect the clustering results. Next, we use DTW as the distance metric to cope with time shifts and different rate variations in ETS data and randomly initialize a cluster centers. By calculating the distance from each time series to each cluster center, it is assigned to the nearest cluster center, and the cluster center is recalculated according to the assignment result,and the process is iterated until the cluster center is stable. We experimented 10 times with different K , and used elbow method to select the optimal number of clusters, and finally determined 12 .

C Full Results

Due to the limited length of the text, we summarize all the experiments in the main text into two parts: the main experiment and the analytical experiment. We categorize and index them in Table 6, 7, 8.

D Limitations

PowerPM is designed for electricity time series modeling, containing about 250M parameters. As a foundation model, although we have provided relatively comprehensive results to verify the model's effectiveness, the model still exsits limitations. In fact, there are various kinds of ETS in the power systems, which contain not only the electricity consumption data generated by human activities, but also the sequence generated by system operation and sensor detection. In this paper, PowerPM only pre-train on load data. In the future, by increasing model parameters and improving model architecture, we will use more kinds of ETS data for training, so that it can capture more complicated ETS semantic information, understand more complex power system operation rules, and provide more complete help for power systems.

E Social Impacts

This paper presents PowerPM as a foundation model for power systems and has been deployed in the real scenario. It focus on demand-side management, grid stability and consumer behavior analysis, providing the possibility to understand and analyze electricity time series. There is no potential ethical risk or negative social impact.

NeurIPS Paper Checklist

1. Claims

Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?

Answer: [Yes]

Justification: In this paper, abstract and introduction accurately reflect the paper's contributions and scope.

Guidelines:

The answer NA means that the abstract and introduction do not include the claims made in the paper.

The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers.

The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.

It is fine to include aspirational goals as motivation as long as it is clear that these goals are not attained by the paper.

2. Limitations

Question: Does the paper discuss the limitations of the work performed by the authors?

Answer: [Yes]

Justification: The limitations of the work are discussed in Appendix D

Guidelines:

The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.

The authors are encouraged to create a separate "Limitations" section in their paper.

The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.

The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on a few datasets or with a few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.

The authors should reflect on the factors that influence the performance of the approach. For example, a facial recognition algorithm may perform poorly when image resolution is low or images are taken in low lighting. Or a speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.

The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.

If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.

While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an important role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.

3. Theory Assumptions and Proofs

Question: For each theoretical result, does the paper provide the full set of assumptions and a complete (and correct) proof?

Answer: [NA]

Justification: The paper does not include theoretical results.

Guidelines:

The answer NA means that the paper does not include theoretical results.

All the theorems, formulas, and proofs in the paper should be numbered and crossreferenced.

All assumptions should be clearly stated or referenced in the statement of any theorems.

The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide a short proof sketch to provide intuition.

Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.

Theorems and Lemmas that the proof relies upon should be properly referenced.

4. Experimental Result Reproducibility

Question: Does the paper fully disclose all the information needed to reproduce the main experimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?

Answer: [Yes]

Justification: The private dataset so one may cannot reproduce the experiments on the private dataset. However, we also use another two public datasets, and the code of our work is fully provided too, which could help you understand our work.

Guidelines:

The answer NA means that the paper does not include experiments.

If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.

If the contribution is a dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.

Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g., in the case of a large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.

While NeurIPS does not require releasing code, the conference does require all submissions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example

(a) If the contribution is primarily a new algorithm, the paper should make it clear how to reproduce that algorithm.

(b) If the contribution is primarily a new model architecture, the paper should describe the architecture clearly and fully.

(c) If the contribution is a new model (e.g., a large language model), then there should either be a way to access this model for reproducing the results or a way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).

(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome to describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.

5. Open access to data and code

Question: Does the paper provide open access to the data and code, with sufficient instructions to faithfully reproduce the main experimental results, as described in supplemental material?

Answer: [Yes]

Justification: The private dataset so one may cannot reproduce the experiments on the private dataset. However, we also use another four public datasets, and the code of our work is fully provided too, which could help you understand our work.

Guidelines:

The answer NA means that paper does not include experiments requiring code.

Please see the NeurIPS code and data submission guidelines ( https://nips.cc/ public/guides/CodeSubmissionPolicy ) for more details.

While we encourage the release of code and data, we understand that this might not be possible, so 'No' is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for a new open-source benchmark).

The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines ( https: //nips.cc/public/guides/CodeSubmissionPolicy ) for more details.

The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.

The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only a subset of experiments are reproducible, they should state which ones are omitted from the script and why.

At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).

Providing as much information as possible in supplemental material (appended to the paper) is recommended, but including URLs to data and code is permitted.

6. Experimental Setting/Details

Question: Does the paper specify all the training and test details (e.g., data splits, hyperparameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?

Answer: [Yes]

Justification: We give a detailed description of our experiment setting in Appendix B.1.

Guidelines:

The answer NA means that the paper does not include experiments.

The experimental setting should be presented in the core of the paper to a level of detail that is necessary to appreciate the results and make sense of them.

The full details can be provided either with the code, in appendix, or as supplemental material.

7. Experiment Statistical Significance

Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?

Answer: [Yes]

Justification: The overall standard derivation is shown in the figure 4.

Guidelines:

The answer NA means that the paper does not include experiments.

The authors should answer "Yes" if the results are accompanied by error bars, confidence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper.

The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).

The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)

The assumptions made should be given (e.g., Normally distributed errors).

It should be clear whether the error bar is the standard deviation or the standard error of the mean.

It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have a 96% CI, if the hypothesis of Normality of errors is not verified.

For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out of range (e.g. negative error rates).

If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.

8. Experiments Compute Resources

Question: For each experiment, does the paper provide sufficient information on the computer resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?

Answer: [Yes]

Justification: We provide information about Experiments Compute Resources such as CPU, GPU, etc. in the Appendix B.

Guidelines:

The answer NA means that the paper does not include experiments.

The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.

The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute.

The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make it into the paper).

9. Code Of Ethics

Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics https://neurips.cc/public/EthicsGuidelines ?

Answer: [Yes]

Justification: All authors reviewed and conducted the NeurIPS Code of Ethics.

Guidelines:

The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.

If the authors answer No, they should explain the special circumstances that require a deviation from the Code of Ethics.

The authors should make sure to preserve anonymity (e.g., if there is a special consideration due to laws or regulations in their jurisdiction).

10. Broader Impacts

Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?

Answer: [Yes]

Justification: Our potential positive societal impacts is discussed in Appendix E.

Guidelines:

The answer NA means that there is no societal impact of the work performed.

If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact.

Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.

The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.

The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.

If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML).

11. Safeguards

Question: Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?

Answer: [No]

Justification: This paper poses no such risks.

Guidelines:

The answer NA means that the paper poses no such risks.

Released models that have a high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.

Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.

We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make a best faith effort.

12. Licenses for existing assets

Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?

Answer: [Yes]

Justification: All the existing assets are properly referenced.

Guidelines:

The answer NA means that the paper does not use existing assets.

The authors should cite the original paper that produced the code package or dataset.

The authors should state which version of the asset is used and, if possible, include a URL.

The name of the license (e.g., CC-BY 4.0) should be included for each asset.

For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.

If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswithcode.com/datasets has curated licenses for some datasets. Their licensing guide can help determine the license of a dataset.

For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.

If this information is not available online, the authors are encouraged to reach out to the asset's creators.

13. New Assets

Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?

Answer: [Yes]

Justification: Our code is provided as a supplement.

Guidelines:

The answer NA means that the paper does not release new assets.

Researchers should communicate the details of the dataset/code/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc.

The paper should discuss whether and how consent was obtained from people whose asset is used.

At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.

14. Crowdsourcing and Research with Human Subjects

Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?

Answer: [NA]

Justification: This paper does not involve crowdsourcing nor research with human subjects. Guidelines:

The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.

Including this information in the supplemental material is fine, but if the main contribution of the paper involves human subjects, then as much detail as possible should be included in the main paper.

According to the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.

15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects

Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?

Answer: [No]

Justification: This paper does not involve crowdsourcing nor research with human subjects. Guidelines:

The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.

Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.

We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution.

For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.

Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper introduces the PowerPM: Foundation Model for Power Systems, which is designed to address the challenges of learning a generic representation of electricity time series (ETS) data in power systems. The model incorporates a temporal encoder and a hierarchical encoder to effectively capture the complex hierarchical structure and temporal dependencies present in ETS data. Additionally, PowerPM utilizes a self-supervised pre-training framework that includes masked ETS modeling and dual-view contrastive learning to enhance its ability to capture temporal dependencies and discrepancies across ETS windows. Overall, the contributions of the paper lie in the development of a sophisticated model that can accurately represent and analyze ETS data in power systems, offering valuable insights for various applications in the field.\n\nSTRENGTHS:\n### Strengths Assessment:\n\n1. **Originality:**\n   - The paper demonstrates originality in its approach to addressing the challenges of modeling electricity time series data in power systems. The combination of a temporal encoder and hierarchical encoder, along with the self-supervised pre-training framework, showcases innovative thinking in capturing temporal dependencies and hierarchical correlations simultaneously.\n   - The use of masked ETS modeling and dual-view contrastive learning in the pre-training stage adds a novel dimension to the model's ability to learn universal representations from ETS data.\n\n2. **Quality:**\n   - The quality of the paper is evident in the thorough description of the PowerPM model, its components, and the experimental results obtained. The model's deployment in real-world scenarios and the achievement of state-of-the-art performance on diverse downstream tasks within private and public datasets reflect the high quality of the research.\n   - The incorporation of extensive experiments, ablation studies, and few-shot experiments provides a robust evaluation of the model's effectiveness and generalization ability across various tasks and domains.\n\n3. **Clarity:**\n   - The paper is well-structured and clearly articulates the motivation, methodology, and results of the PowerPM model. The descriptions of the temporal encoder, hierarchical encoder, and the self-supervised pre-training framework are presented in a coherent manner, making it easy for readers to understand the technical aspects of the model.\n   - The inclusion of figures, tables, and detailed explanations aids in clarifying complex concepts such as the model analysis, ablation study, and model scale evaluation.\n\n4. **Significance:**\n   - The significance of the paper lies in its contribution to advancing the modeling of electricity time series data in power systems. By introducing the PowerPM model, which effectively captures temporal dependencies and hierarchical correlations, the research offers a valuable tool for enhancing economic efficiency and promoting low-carbon principles in power systems.\n   - The model's superior performance on diverse downstream tasks, its generalization ability across different datasets, and the economic benefits generated in real-world deployments underscore the practical significance of the research in the field of power systems modeling.\n\nOverall, the paper excels in originality, quality, clarity, and significance, making a substantial contribution to the domain of electricity time series data modeling in power systems.\n\nWEAKNESSES:\n### Weaknesses Assessment:\n\n1. **Limited Comparison with State-of-the-Art Models:**\n   - While the paper highlights the superior performance of PowerPM over baseline models, there is a lack of comparison with the most recent state-of-the-art models in the field of electricity time series data analysis. Including comparisons with cutting-edge models could provide a more comprehensive evaluation of PowerPM's performance [T1].\n\n2. **Insufficient Discussion on Model Interpretability:**\n   - The paper could benefit from a more in-depth discussion on the interpretability of the PowerPM model. Providing insights into how the model captures and represents temporal dependencies and hierarchical correlations in ETS data could enhance the understanding of its inner workings and decision-making processes.\n\n3. **Limited Exploration of Hyperparameters and Sensitivity Analysis:**\n   - The paper lacks a detailed exploration of hyperparameters and sensitivity analysis for the PowerPM model. Including a thorough investigation of the impact of hyperparameters on model performance and conducting sensitivity analysis could provide valuable insights into the robustness and stability of the model across different settings.\n\n4. **Scalability and Efficiency Considerations:**\n   - The paper could further address scalability and efficiency considerations of the PowerPM model, especially when applied to large-scale datasets or real-time applications. Discussing the computational requirements, training time, and potential bottlenecks in scaling the model could help in understanding its practical feasibility in industrial settings.\n\n5. **Limited Discussion on Ethical and Societal Implications:**\n   - The paper could expand its discussion to include ethical and societal implications of deploying the PowerPM model in real-world power systems. Addressing issues related to data privacy, fairness, and potential biases in the model's predictions could enhance the overall impact and relevance of the research in a broader context.\n\n### Suggestions for Improvement:\n\n1. **Incorporate Comparison with State-of-the-Art Models:**\n   - Conduct a thorough comparison with the latest state-of-the-art models in electricity time series analysis to provide a more comprehensive evaluation of PowerPM's performance and highlight its competitive advantages.\n\n2. **Enhance Model Interpretability Analysis:**\n   - Include a section dedicated to explaining how the PowerPM model interprets and processes ETS data, shedding light on its decision-making processes and enhancing the transparency of the model.\n\n3. **Conduct Hyperparameter Tuning and Sensitivity Analysis:**\n   - Perform a detailed hyperparameter tuning process and sensitivity analysis to understand the impact of key parameters on model performance and ensure robustness across different scenarios.\n\n4. **Address Scalability and Efficiency Concerns:**\n   - Discuss the scalability and efficiency aspects of the PowerPM model, including computational requirements, training time optimization strategies, and considerations for real-time deployment in power systems.\n\n5. **Expand Discussion on Ethical and Societal Implications:**\n   - Include a section on the ethical and societal implications of deploying the PowerPM model, addressing issues of fairness, bias, and privacy to ensure responsible and ethical use of the model in practical applications.\n\nQUESTIONS:\n### Questions and Suggestions for the Authors:\n\n1. **Clarification on Model Interpretability:**\n   - Can the authors provide more insights into how the PowerPM model interprets and captures temporal dependencies and hierarchical correlations in electricity time series data? Understanding the interpretability of the model can enhance its transparency and trustworthiness in real-world applications.\n\n2. **Explanation on Hyperparameter Selection:**\n   - Could the authors elaborate on the rationale behind the selection of specific hyperparameters for the PowerPM model? Providing details on the hyperparameter tuning process and its impact on model performance would offer valuable insights into the model's robustness.\n\n3. **Discussion on Generalization to Unseen Domains:**\n   - How does the PowerPM model generalize to unseen domains or datasets outside the ones used in the experiments? Understanding the model's ability to adapt to new data distributions and scenarios is crucial for assessing its practical utility in diverse real-world applications.\n\n4. **Scalability and Deployment Considerations:**\n   - What are the scalability considerations for deploying the PowerPM model in large-scale power systems? How does the model handle real-time data processing and what are the potential challenges in scaling it for industrial applications?\n\n5. **Ethical and Fairness Implications:**\n   - Have the authors considered the ethical implications of using the PowerPM model in power systems, particularly in terms of data privacy, fairness, and potential biases? Addressing these concerns can ensure responsible and unbiased deployment of the model in practice.\n\n6. **Future Directions and Extensions:**\n   - Are there any plans to extend the PowerPM model or explore new research directions based on the current findings? Discussing potential future developments or applications of the model can provide insights into the ongoing research trajectory in the field of power systems modeling.\n\n7. **Comparison with Latest Research:**\n   - Have the authors considered comparing the PowerPM model with the most recent state-of-the-art models in electricity time series analysis? Including such comparisons can strengthen the paper's contribution and highlight the advancements made by PowerPM in the field.\n\nAddressing these questions and suggestions can enhance the clarity, robustness, and applicability of the PowerPM model, providing valuable insights for both the research community and practical stakeholders in the power systems domain.\n\nLIMITATIONS:\n### Assessment of Limitations and Societal Impact Considerations:\n\n1. **Limitations Addressed:**\n   - The authors have discussed the limitations of their work in Appendix F, which is a positive step towards transparency and acknowledging the constraints of the research. This demonstrates a willingness to reflect on the scope and potential weaknesses of the study.\n\n2. **Societal Impact Considerations:**\n   - The paper lacks a comprehensive discussion on the potential negative societal impacts of deploying the PowerPM model in real-world power systems. While some ethical considerations are mentioned, a more thorough exploration of broader societal implications is needed.\n\n### Suggestions for Improvement:\n\n1. **Enhanced Limitations Section:**\n   - The authors could consider expanding the limitations section in the main body of the paper rather than relegating it to an appendix. This would ensure that readers engage with the limitations more prominently and understand the boundaries of the research.\n\n2. **Broader Societal Impact Analysis:**\n   - To address potential negative societal impacts, the authors should conduct a detailed analysis of how the deployment of the PowerPM model could affect various stakeholders, including issues related to fairness, privacy, and bias. Providing mitigation strategies for these impacts would demonstrate a proactive approach to responsible research.\n\n3. **Incorporate Ethical Frameworks:**\n   - Utilize established ethical frameworks or guidelines to evaluate the ethical implications of the PowerPM model in power systems. This could involve considering principles such as fairness, accountability, transparency, and privacy to ensure ethical deployment and minimize negative societal consequences.\n\n4. **Engage with Stakeholders:**\n   - Engage with relevant stakeholders, such as power system operators, regulators, and community representatives, to gather diverse perspectives on the potential societal impacts of implementing the PowerPM model. Incorporating feedback from stakeholders can enrich the discussion on societal implications.\n\n5. **Mitigation Strategies:**\n   - Propose concrete mitigation strategies for addressing any identified negative societal impacts of the PowerPM model. This could include mechanisms for monitoring model performance, ensuring fairness in decision-making processes, and enhancing transparency in model deployment.\n\nBy addressing these suggestions, the authors can strengthen the ethical foundation of their research, demonstrate a commitment to responsible innovation, and contribute to a more comprehensive understanding of the societal implications of deploying advanced models like PowerPM in practical settings.", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2024-07-11"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nA pre-trained LLM named PowerPM is proposed for modeling Electricity Time Series (ETS) data. PowerPM combines a temporal encoder for capturing temporal patterns and a hierarchical encoder for understanding hierarchical correlations. PowerPM also employs a self-supervised pre-training strategy that incorporates masked ETS modeling and dual-view contrastive learning, enhancing the model's ability to learn from the intricacies of ETS data. \n\nPowerPM is specifically designed for power systems, especially for the demand side, making it unique among other general models. This paper claims that deployment of PowerPM in Zhejiang Power Grid had led to significant economic benefits, showing the practical value of the model in real-world settings.\n\nSTRENGTHS:\nThis article proposes a model for load time series forecasting on multi-hierarchy demand side and pre-trains it on a large amount of real data. It shows good performance and strong generalizability in experiments. Furthermore, the model is applicable in the real world.\nFor multiple pre-training challenges on multi-hierarchy load time series data, this method integrates corresponding solutions together.\n\nWEAKNESSES:\nSince the pre-training dataset is comprehensive, the work lacks an analysis on the effectiveness provided by the dataset. In other words, readers may not understand if the dataset works a lot or the proposed method.\nThe writing and explanation in this paper need to be improved.\nThis paper involves extensive training, but details are largely missing from the current paper.\n\nQUESTIONS:\nThe writing lacks sufficient detail and explanation, resulting in quite some questions.\nFor example, what are the computational resources used? The provided materials do not include codes related to the proposed pre-training method. No information about whether the pre-trained model or the pre-training dataset will be released to make the work transparent.\n\nLIMITATIONS:\nThis works involves load forecasting for power system and individual users in real-world applications, which would affect power system operation, its safety, and societal impact as a whole. These should be discussed.", "TITLE": "Official Review", "RECOMMENDATION": 3, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2024-07-13"}

comments: SUMMARY:
This paper proposed a foundation model, PowerPM, to model electricity time series data, providing a large-scale off-the-shelf model for power systems. PowerPM consists of a temporal encoder and a hierarchical encoder with a self-supervised pretraining framework. The authors have tested PowerPM in Demand-side Management, Grid Stability, and Consumer Behavior Analysis, showing their advantages against other common time series LLM models, such as GPT4, TimeLLM, etc.

STRENGTHS:
1. They have tested their work with real data and showing competitive results;
2. It is the first model that considers temporal dependency and hierarchical dependency simultaneously;
3. Comprehensive model analyses.

WEAKNESSES:
The writing should be improved, several places are missing space between words.
It could be improved or make the paper more convincing if the author could compare their framework with some state-of-art techniques. However, it is understandable that the research question/application objectives are more comprehensive in this paper.

QUESTIONS:
1. For power systems, have you considered the power systems constraints, such as power network structures, physic laws of Kirchhoff's circuit laws?
2. With renewable energy sources, have you considered of including weather time series data along with ETS data?

LIMITATIONS:
The authors has discussed the limitations of their work.
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 3
DATE: 2024-07-13

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper learns a generic representation of electricity time series data. The proposed PowerPM model is composed of a temporal encoder and a hierarchical encoder.\n\nSTRENGTHS:\nThe results shown in the table exhibit good numerical results of the proposed model.\n\nWEAKNESSES:\nIt is unclear where the performance gain comes from. And the proposed architecture is a combination of several previous techniques, so it is hard to identify the technical contributions.\n\nMoreover, it is not well motivated on the usage of developed models for many power system tasks. As far as the reviewer is aware of, current statistical methods and machine learning-based methods can already give good load forecasting results. The real challenges come from noisy data inputs, incomplete number of features, or small-region level forecasting. Yet this paper only looks into a very general forecasting problem, and does not reveal the key challenges and special properties of energy forecasting.\n\nThe grid stability notion is quite misleading. In standard power system tasks, stability notion is with respect to either frequency or voltage, and is more related to the transient states of the systems. While in this paper, the stability is more like a phenomenon or behavior analysis. \n\nIn addition, the paper needs to discuss the computational costs related to both training and inference, as utility and grid operators normally are not equipped with enough computation capabilities to use large foundation models.\n\nQUESTIONS:\nCan the authors elaborate the settings of the freeze version?\n\nWhat are the effects of contrastive learning?\n\nThe figure 4 looks like a software snapshot. Can the authors explain the major information conveyed in this figure?\n\nLIMITATIONS:\nThe paper describes techniques limited to energy forecasting tasks, and it is not clear if the methods can be applied to other tasks.", "TITLE": "Official Review", "RECOMMENDATION": 3, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 2, "DATE": "2024-07-13"}