A Novel Criterion of Electricity Price Forecast for
Demand-side Responses Participating in theElectricity Market
Sinan Cai, Masahiro Mae, Ryuji Matsuhashi
Department of Electrical Engineering & Information Systems
The University of Tokyo
Tokyo, Japan
cai@enesys.t.u-tokyo.ac.jp
ORCID: 0000-0002-7536-0620

Abstract—This paper proposes a novel criterion for evaluating
electricity price forecasting results for demand-side responses(DRs) who participate in the electricity market. Generally, theDR needs to predict the market price and arrange its bidding
and operation schedule according to the forecast result. The
mean-square-error (MSE) or the R-squared coefficient is used
for evaluating the forecast result conventionally. However, it isshown in this paper that a forecast result with a good MSE or
R-squared value is not necessarily more beneficial for the DRs.
Instead, the proposed novel criterion can reflect the influence ofdifferent forecast results on the DR’s market revenue and help
the DR identify which forecast result is better regarding economic
benefits. The proposed criterion emphasizes the accuracy of
predicting the timing of the price peaks and dips, which ismore crucial information to the DRs than the total numerical
forecast precision over time. Results comparing different forecast
methods on the clearing price of the day-ahead energy wholesalemarket in Japan, the JEPX spot market, are reported. The better
forecast method for the DR is identified by the proposed criterion
successfully.
Index Terms—Demand-side Responses, Electricity Market,
Price Forecast, ARIMA models, Deep Learning

I. I NTRODUCTION
The electricity price has been an important research topic
in power systems long before the deregulation of the power
market. Models to calculate the price on an hour-by-hour, busby-bus level were developed based on future regional demand
estimation and regional supply optimal dispatching [1]. The
models aim at system cost minimization and are very detailed
to include all the crucial components in the system since
all the information is available. With the assumption of a
low uncertainty environment and no extra exercisable market
power, one can easily obtain the electricity price through themodel.
The deregulated market environment has replaced the traditional monopolistic scheme in the electricity industry inmany countries. Except for the direct bilateral contracts, theThis work is conducted by the Social Cooperation Research Departmentsof Power System Innovation Realization with Fuji Electric Co., Ltd. in
Collaborative Research Organization for Comprehensive Energy Sciences(CROCES) at the University of Tokyo.
979-8-3503-8174-0/24/$31.00 ©2024 IEEE

electricity price is mostly determined by pool trading, where
the generation side and consumer side submit bids respectively
for selling and purchasing electricity [2]. The electricity prices
might rise or drop excessively depending on the accepted bids
in the pool when the market operator clears the market. More
importantly, not only the traditional generators and powercompanies but also the demand-side resources such as DRs
can participate in the market.
As the competitive framework has been introduced into theelectricity industry and market, a price model with detailed
components of all the resources’ characteristics and operation
behaviors becomes impossible since no entity could have all
the information on every individual market participant. The
price can only be known when the market is cleared. Moreover,due to the non-storable nature of electrical energy, the markets
are usually in a day-ahead or hour-ahead scheme to ensurethat during real-time operation balance between demand andsupply can be maintained.
The market price is essential to the market participants’decision-making process such as bidding strategy optimizationand self-operation scheduling. As a result, short-term priceforecasting has emerged as a heated research topic in thepast two decades. The price curve in actual electricity markets
exhibits a far more complex structure than the load curve suchas nonconstant mean and variance, high level of volatility, andunexpected spikes [3]. The prediction task can be extremelycomplicated and many techniques have been adopted to challenge the task.
The price forecasting methodologies can be generally classified into three categories: the game theory models, thesimulation models, and the time-series models [4]. For anormal market participant, since the detailed information ofother participants in the market is usually unclear, time-seriesmodels focusing on the past behavior of the dependent variables are a more cost-effective approach. The auto-regressive
integrated moving average (ARIMA) model is a common
stochastic time-series analysis method and predicts future
values based on the historic values without extra information
input [5]. The application of the ARIMA model in electricity

price forecasting can be found in [6], while the seasonal autoregressive integrated moving average (SARIMA) model isused to consider the seasonal trend of Indian electricity market
price in [7]. Likewise, the utilization of Generalized Autoregressive Conditional Heteroskedasticity (GARCH) model isreported in [8]. The stochastic time-series forecast can befurther improved by adopting the wavelet transform to preprocess the historical price data [9].
The rapid development of machine learning has also inspired the forecast models based on deep learning neuralnetworks. These models are sorely data-based and can detectand extract correlations from historical time-series data autonomously. Common back propagation neural network is usedin [10]–[12], while the application of the more sophisticatedconvolution neural network (CNN) and long short-term memory (LSTM) structure can be found in [13]–[15]. Moreover,the combination of stochastic time-series analysis and neuralnetwork models is reported in [16], [17].
However, different forecast techniques might perform differently in different markets and under different situations, especially when most of the real-world electricity market is still
evolving with new mechanisms being set up to address current
system problems like increasing and intermittent renewablesource generation. With various existing forecast techniques
to choose from, a market participant would require a criterion
to help them select the most suitable forecast techniques. So
far, most of the existing studies only focus on the numericalprecision of every forecast point, which does not necessarily
stand for a good forecast. For example, in the clinical field,
the overall trajectory of a patient’s blood glucose is more
important than the numerical value [18]. The capacity of
the DRs is usually small and their operation behaviors aredifferent from the conventional resources in the system. The
business model of DRs usually relies on aggregation or change
in lifestyle, making them very sensitive to economic profits.
For this reason, a different criterion of forecast evaluation isneeded. The core of this paper is not to compare the existing
forecast methods directly but to provide a way to distinguishthe performance of given forecast results from an economic
perspective.
The rest of this paper is organized as follows: the potential
problem of the conventional forecast result evaluation criterion
is identified in Section II. Section III introduces the proposednovel criterion. The simulation with the real historical priceof the day-ahead energy market in Japan is demonstrated tovalidate the proposed criterion in Section IV, followed by theconclusion in Section V.
II. D EFICIENCY OF C ONVENTIONAL F ORECAST R ESULTE VALUATION
The mean-absolute-error (MAE) and the mean-square-error(MSE) are commonly used criteria for forecast result evaluation, both evaluate the forecast result by summing the forecast
errors of all the forecast points. The smaller the sum of
the errors, the better the result is. Likewise, the R-squared
coefficient converts the sum of forecast errors to a coefficient

below 1. The forecast result is considered good if the Rsquared coefficient is close to 1.
The methods are very straightforward and intuitive, as agood forecast result is expected to be as close as possible tothe actual values. Generally, this is the case for market priceforecasts, whether in stock markets or electricity markets. The
market participants have the motivation to keep track of theprices at every time step so that better strategies can be taken.
For example, a generation company or an electricity retailer
can schedule their day-ahead generation plan or electricity
purchase/sell plan according to the forecast of the energy
whole sell market price. The plan covers the whole day,therefore, a precise forecast on every time step is desired.
However, the case is different for the DRs, mainly for
two reasons. First, the DRs are unlikely to participate in themarket during a long time span. Typical DRs are household
equipment such as water heat pumps, electric vehicles (EVs),or air conditioners. Those equipment have their own purposesand cannot always respond to the system’s demand. Even forbattery storage systems, the charging and discharging time
only lasts for a few hours. Second, the capacity of a DR isgenerally very small compared to other resources in the market
even when aggregated. In most cases, the DR will act as a
price taker in the market and does not affect the market pricedirectly like big generation or consumption resources.
A detailed example is given in Fig. 1. The blue line denotes
the actual clearing price of a single-price market and thetwo forecast results are shown in red and yellow respectively.
At first glance, Prediction 2 is clearly the better prediction
with a R-squared value of 0.87, while the R-squared value ofPrediction 1 is only 0.19. To maximize the profit, the DRs arelikely to consume energy when the price is low and output
energy when the price is high. In this case, assuming the
DR is a battery system that charges at the lowest price anddischarges at the highest price according to the forecast results,
the operation schedule and the profit of the DR are listed
in Table I. In spite of a better R-squared value, Prediction
2 wrongly forecast the timing of the highest and the lowest
price, leading to 0.62 profit of the DR. On the other hand, evenwith a worse R-squared value, Prediction 1 is correct about
the timing of the highest and the lowest price, hence full profit
can be obtained.
TABLE IDR’ S O PERATION S CHEDULE AND P ROFIT
ActualPrediction 1
Prediction 2

Charge Time551Discharge Time
15
15
19

Profit
2
2
0.62

R2
1
0.19
0.87

To sum up, the critical market price information for DRs
is to find out the timing of the peak and dip in the price,so that they can arrange their schedule and participate in themarket when the peak or dip comes. This phenomenon is
first pointed out in [19], where aggregated EVs participate
in the frequency regulation market during their parking time.

Fig. 1. Example of price forecast result comparison.

The EV aggregators gain 88.3% of the theoretical maximum
revenue with a ’poor’ prediction whose R-squared value isonly 0.2. The conventional way of forecast result evaluation
sorely based on the sum of the error of all the forecast points
is not appropriate due to the inherent characteristics of DRs.
When evaluating forecast results for DRs, the accuracy of
the timing of the price peak and dip should be taken into
consideration as well.

while F and A stand for the forecast result and the actual pricerespectively. The basic idea of ET A is to evaluate whether theforetasted local maximum and minimum timing is correct in
the actual price, the larger the ET A is, the better the forecast
result. Note that the numerator part of the ET A alone canbe used for evaluation as well. The denominator part is used
to scale the ET A down to a value not bigger than 1, and itis the same for all forecast results. When the timing of thelocal maximum and the local minimum of the forecast resultis exactly the same as those of the actual price, the ET A
reaches 1, indicating a perfect forecast in terms of the ET A.
A calculation example using the price and forecast results
shown in Fig. 1 is given here. For the denominator part, theactual price’s local maximum and local minimum index are
15 and 5, respectively. Accordingly, the denominator part of
the ET A is Y (15) − Y (5) = 2 − 0 = 0. For Prediction 1, thelocal maximum and the local minimum index are identical tothe actual price, thus the numerator part equals 2 as well andthe ET A is 1. On the other hand, Prediction 2 has two local
maximums(at 2 and 19) and two local minimums(at
1 and
P
8).
The
numerator
part
can
be
calculated
as
Y
(i) −
i=2,19P
i=1,8 Y (i). Despite a R-squared value of 0.87, the ET A
drops to 0.3, indicating that the accuracy of the extremum
timings is low and the forecast result is considered poor for theDRs. The above calculation process is summarized in Table II.

III. T HE N OVEL C RITERION FOR D EMAND - SIDER ESPONSE
The most intuitive and accurate way to evaluate a forecast
result from a profit perspective is to directly compare theactual profit from the market as in [19]: the optimal operation
schedule and market bidding strategy was first determined
to maximize the EV aggregators’ profit and then the actual
profit was calculated by market simulation with the actual
market price. Different forecasts resulted in different operation
strategies, which in turn led to different profit losses.
However, such an approach has a few drawbacks:• The operation strategy of the DRs must be determined
and modeled in advance. Meanwhile, the capacity andoutput power also need to be fixed in order to perform
operation optimization and market simulation for profit
calculation.
• The result of direct profit comparison is only valid forthe specific case.
Therefore, a general way to evaluate the forecast results
can be more helpful and informative for the DRs, especiallywhen the business model is not decided. The proposed novel
criterion, Extremum Timing Accuracy (ET A), is defined in(1):PP
F Y (i) −F Y (i)
Pi∈LM inET A = Pi∈LM ax
(1)
Y
(i)
−
A
A
i∈LM axi∈LM in Y (i)Y is the actual price in the time series. Lmax and Lmin are
the indexes of the local maximum and local minimum point,

TABLE IIETA C ALCULATION

Prediction 1
Prediction 2

Lmax
15
2
19

Y (i)
2
0.4
1.3

Lmin
5
1
8

Y (i)
0
0.7
0.4

ET A
2−0
=1
2−0
(0.4+1.3)−(0.7+0.4)
= 0.3
2−0

IV. M ARKET S IMULATIONThe example given in Section III is an exaggerated situation to demonstrate the difference between the proposed
criterion ET A and the conventional R-squared value. In thissection, three given forecast results are to be evaluated. A
DR optimizes its operation and bidding schedule based on thegiven forecast results respectively. The market simulation is
performed to compare the profits and validate the evaluationin MATLAB.

A. DR Model
The DR is a battery storage system that intends to purchasecharging power from the energy spot market and sell the powerback to the market by discharging power to gain profit. The
daily operation schedule optimization problem is described in(2):

maximizeState(t)Pmax

X

P rice(t)State(t)

(2a)

t

subject to SOC(t) = SOCinitial − Pmax ∆tXState(t),a daily basis by Mixed Integer Linear Programming, and the
profit is calculated according to the optimization result.
P rof it = Pmax ∆tX

Y (t)State(t)

(3)

t

t

(2b)
0 ≤ SOC(t) ≤ 1,
X−
State(t) ≤ n if State(t) < 0

(2c)
(2d)

t

The objective function (2a) is to maximize the DR’s one-day
profit. P rice(t) is the given forecast market price and Pmax
is the maximum power output of the DR. State(t) ∈ [−1, 1]
is the charging and discharging schedule to be optimized. The
state-of-charge (SOC) of the DR is maintained by constraint(2b) and (2c). The number of charging cycles is limited by (2d)to prevent massive battery deterioration, and n is the numberof charging cycles allowed per day. ∆t is the length of themarket time-step.
B. Simulation Result
The target market is the energy wholesale market in Japan:
the JEPX spot market. The JEPX spot market is a day-ahead
single-price market with a 30-minute time step. The market
participants need to submit the bid by 10:00 one day before[20].
Two of the forecast results to be evaluated are obtained bya SARIMA and a Machine Learning model. The SARIMA
model is derived by the Econometrics Toolbox in MATLAB.
The Machine Learning model is developed by the Japan
Weather Association, utilizing historical price data, weather
data, and calendar data [21]. The price data for June 2023
is used in this paper. The actual and two forecast prices are
depicted in Fig. 2. The other forecast is to simply use theday-ahead price as the forecast, namely the Yesterday model.
18
16
14
12
10
8
6
4
2
0
2023-06-01

2023-06-07

2023-06-13

2023-06-19

2023-06-25

2023-07-01

Fig. 2. Price Data

The DR acts as a price-taker in the market. Therefore theprofit can be calculated as in (3). State(t) is the optimized
charging and discharging schedule in (2). The DR’s capacity is
10 MWh with a maximum power output of 10MW. 2 charging
cycles are allowed per day. The optimization is executed on

The simulation result is shown in Fig. III. The Oracle caseis a benchmark case standing for the perfect forecast andthe profit of the DR reaches the theoretical upper limit. The
Machine Learning forecast has the highest R-squared valueof 0.5 and can be considered the best forecast result among
the three results conventionally. However, the ET A of theMachine Learning forecast result is only 0.54, close to that of
the Yesterday forecast. The R-squared value of the SARIMA
forecast is 0.34, but the ET A reaches 0.69, indicating the
SARIMA forecast should be the most profitable forecast result.
The outcome of the actual profit calculation agrees with the
ET A evaluation. The profit of the SARIMA forecast is 356.1
thousand JPY, reaching 81% of the theoretical limit. In contrast, despite the high R-squared value, the Machine Learning
forecast obtained around 260 thousand ¥. On the other hand,
the Yesterday forecast’s R-squared value is extremely poor,
but the actual profit is close to the Machine Learning forecast
since their ET A values are close too.
TABLE IIIS IMULATION R ESULT

Forecast

R2

ET A

OracleMachine Learning
SARIMA
Yesterday

1
0.55
0.34
0.08

1
0.54
0.69
0.53

Profit(Thousand JPY)
391.63265.36318.11249.07

Profit Ratio
100%
67.76%
81.23%
63.60%The detailed operation schedule on June 10th, 2023 isshown in Fig. 3. The positive blue bar indicates selling energy
by discharging, while the negative one indicates purchasingenergy to charge. The battery SOC is illustrated by the redline. The Machine Learning forecast seems to be closer to theactual price, especially at around 6:00 to 15:00. However, since
the price is dipping during this period, the DR is expected
to purchase energy. As long as the trend of the price dipping
trajectory is reflected by the forecast, the DR will be optimized
to purchase energy no matter how much the predicted priceis. In the meantime, the Machine Learning forecast introduces
unnecessary dynamics at 8:00, 10:30, and 22:30 regardlessof the high R-squared value, hence the operation bidding
schedule strays away from the Oracle case. Contrariwise, theSARIMA forecast reflects the tendency of the price dipping
at noon and consequently, the operation schedule is close to
the Oracle case and more profitable.
V. C ONCLUSIONSThis paper proposes a novel criterion of electricity priceforecast, ET A, for the DRs. Unlike traditional resources, theDRs generally do not participate in the market for the wholePrice [JPY/kWh]Price Data
12
10
8
6
4
2

ActualMachine Learning
SARIMA
Yesterday

2023-06-10 00:00

2023-06-10 06:00

0.5

SOC

0

2023-06-10 06:00

2023-06-10 12:00

Time

2023-06-10 18:00

Machine Learning Bid

0

1

0

0.5

SOC

0.5

-0.5

-1
2023-06-10 00:00

2023-06-10 06:00

2023-06-10 12:00

Time

2023-06-10 18:00

SARIMA Bid

1

0

1

0

0.5

SOC

0.5

-0.5

-1
2023-06-10 00:00

2023-06-10 06:00

2023-06-10 12:00

Time

2023-06-10 18:00

Yesterday Bid

1

0

1

0.5
0

0.5

SOC

Bid [kWh]

1

-0.5

1

Bid [kWh]2023-06-10 18:00

0.5-12023-06-10 00:00

Bid [kWh]Time

Oracle Bid

1

Bid [kWh]2023-06-10 12:00

-0.5

-1
2023-06-10 00:00

2023-06-10 06:00

2023-06-10 12:00

Time

2023-06-10 18:00

0

Fig. 3. Detailed Operation Schedule on June 10th, 2023

time, hence a forecast result recognized by conventional Rsquared value or MSE is not necessarily beneficial for the DRs.
The proposed criterion ET A is designed to emphasize thetiming of the price peaks and dips, which is the most criticalprice information for the DRs. With the help of the proposed
criterion, the DRs can easily identify the most profitable forecast under different situations without developing a detailed
operation model and performing simulations to calculate andcompare the actual profit.
The future research direction includes the combination of
ET A with the R-squared value. When the DR is participating in multiple markets simultaneously, the accuracy of theforecast value becomes more important because the DR mightneed to compare and choose which market to participate. In
such a situation, evaluation merging both ET A and the Rsquared value seems more reasonable and appropriate. Anotherpotential future research direction is new forecast algorithm
and technique development based on ET A. For example, thenumerator part of ET A can be used as the loss function inmachine learning to design a forecast model producing high
ET A forecast results.
R EFERENCES[1] A. Angelus, “Electricity price forecasting in deregulated markets,” The
Electricity Journal, vol. 14, no. 3, pp. 32–41, 2001.

[2] F. C. Schweppe, M. C. Caramanis, R. D. Tabors, and R. E. Bohn, Spotpricing of electricity. Springer Science & Business Media, 2013.
[3] J. P. d. S. Catalão, S. J. P. S. Mariano, V. Mendes, and L. Ferreira, “Shortterm electricity prices forecasting in a competitive market: A neuralnetwork approach,” electric power systems research, vol. 77, no. 10, pp.
1297–1304, 2007.
[4] A. M. González, A. S. Roque, and J. Garcı́a-González, “Modeling andforecasting electricity prices with input/output hidden markov models,”IEEE Transactions on Power Systems, vol. 20, no. 1, pp. 13–24, 2005.
[5] G. E. Box, G. M. Jenkins, G. C. Reinsel, and G. M. Ljung, Time series
analysis: forecasting and control. John Wiley & Sons, 2015.
[6] T. Jakaša, I. Andročec, and P. Sprčić, “Electricity price forecasting—arima model approach,” in 2011 8th International Conference on
the European Energy Market (EEM). IEEE, 2011, pp. 222–225.
[7] P. Rajan and K. V. Chandrakala, “Statistical model approach of electricity price forecasting for indian electricity market,” in 2021 IEEE Madras
Section Conference (MASCON). IEEE, 2021, pp. 1–5.
[8] R. C. Garcia, J. Contreras, M. Van Akkeren, and J. B. C. Garcia, “Agarch forecasting model to predict day-ahead electricity prices,” IEEE
transactions on power systems, vol. 20, no. 2, pp. 867–874, 2005.
[9] A. J. Conejo, M. A. Plazas, R. Espinola, and A. B. Molina, “Dayahead electricity price forecasting using the wavelet transform and arimamodels,” IEEE transactions on power systems, vol. 20, no. 2, pp. 1035–
1042, 2005.
[10] D. Singhal and K. Swarup, “Electricity price forecasting using artificial
neural networks,” International Journal of Electrical Power & Energy
Systems, vol. 33, no. 3, pp. 550–555, 2011. [Online]. Available:
https://www.sciencedirect.com/science/article/pii/S0142061510002231[11] D. Keles, J. Scelle, F. Paraschiv, and W. Fichtner, “Extended forecast
methods for day-ahead electricity spot prices applying artificial neuralnetworks,” Applied energy, vol. 162, pp. 218–230, 2016.
[12] D. Wang, H. Luo, O. Grunder, Y. Lin, and H. Guo, “Multi-step ahead
electricity price forecasting using a hybrid model based on two-layer
decomposition technique and bp neural network optimized by fireflyalgorithm,” Applied Energy, vol. 190, pp. 390–407, 2017.
[13] P.-H. Kuo and C.-J. Huang, “An electricity price forecasting model by
hybrid structured deep neural networks,” Sustainability, vol. 10, no. 4,
p. 1280, 2018.
[14] K. Wang, M. Yu, D. Niu, Y. Liang, S. Peng, and X. Xu, “Short-term
electricity price forecasting based on similarity day screening, twolayer decomposition technique and bi-lstm neural network,” Applied Soft
Computing, vol. 136, p. 110018, 2023.
[15] F. H. Mohammadreza Heidarpanah and M. Fazeli, “Daily electricity
price forecasting using artificial intelligence models in the iranian electricity market,” Energy, vol. 263, p. 126011, 2023. [Online]. Available:
https://www.sciencedirect.com/science/article/pii/S0360544222028973[16] N. Amjady and F. Keynia, “Day ahead price forecasting of electricity
markets by a mixed data model and hybrid forecast method,” International Journal of Electrical Power & Energy Systems, vol. 30, no. 9, pp.
533–546, 2008.
[17] J. C. R Filho, C. M. Affonso, and R. C. Oliveira, “Energy priceforecasting in the north brazilian market using nn-arima model andexplanatory variables,” in 2014 IEEE Symposium on Computational
Intelligence for Engineering Solutions (CIES). IEEE, 2014, pp. 171–175.
[18] I. Fox, L. Ang, M. Jaiswal, R. Pop-Busui, and J. Wiens, “Deepmulti-output forecasting: Learning to accurately predict blood glucose
trajectories,” in Proceedings of the 24th ACM SIGKDD international
conference on knowledge discovery & data mining, 2018, pp. 1387–
1395.
[19] S. Cai and R. Matsuhashi, “Model predictive control for ev aggregators
participating in system frequency regulation market,” IEEE Access,vol. 9, pp. 80 763–80 771, 2021.
[20] Japan Electric Power eXchange Guide, Japan Electric Power Exchange,
2019. [Online]. Available: https://www.jepx.jp/electricpower/outline/
pdf/Guide 2.00.pdf[21] “Price forecast (jepx forecast),” Japan Weather Association.
[Online]. Available: https://www.jwa.or.jp/service/weather-and-data/
weather-and-data-03/