# Wholesale Bidding Approaches of an Autonomous Trading Agent in Electricity Markets

Serkan Özdemir, Rainer Unland

► To cite this version:

Serkan Özdemir, Rainer Unland. Wholesale Bidding Approaches of an Autonomous Trading Agent in Electricity Markets. 3rd and 4th International Conference on Smart Energy Research (SmartER Europe 2016 and 2017), Feb 2016, Essen, Germany. pp.90-104, 10.1007/978-3-319-66553-5_7. hal-01691202

HAL Id: hal-01691202

https://inria.hal.science/hal-01691202

Submitted on 23 Jan 2018

HAL is a multi-disciplinary open access
archive for the deposit and dissemination of sci-
entific research documents, whether they are pub-
lished or not. The documents may come from
teaching and research institutions in France or
abroad, or from public or private research centers.

L'archive ouverte pluridisciplinaire HAL, est
destinée au dépôt et à la diffusion de documents
scientifiques de niveau recherche, publiés ou non,
émanant des établissements d'enseignement et de
recherche français ou étrangers, des laboratoires
publics ou privés.

<figure><img src="image_1.png" /></figure>

Distributed under a Creative Commons Attribution 4.0 International License
# Wholesale Bidding Approaches of an Autonomous Trading Agent in Electricity Markets

Serkan Özdemir and Rainer Unland

DAWIS, University of Duisburg-Essen, Schützenbahn 70, 45127 Essen
{serkan.ozdemir,rainer.unland}@icb.uni-due.de

**Abstract.** Energy transition policies, including Energiewende in Germany, plan to replace conventional power plants with renewable energy resources thanks to advances on solar and wind technologies. However, shifting from conventional production to highly volatile renewable energy production will bring a number of challenges as well. One of those challenges is price forecasting accuracy in renewable energy driven markets. Intermediary power actors, e.g., retailers, suffer from that problem at most, due to their load-balancing role (financially). This work presents a number of electricity price forecasting approaches, and aims to refine the forecasting error, using additional drivers such as weather observation data. The proposed bidding approaches have been tested in our broker agent which competes in Power Trading Agent Competition (Power TAC). In detail, our model firstly makes price predictions out of historical market clearing prices, using a hybrid model. This model alters a seasonal regression model by replacing the aged terms with a belief function. Then, those predicted prices are reassessed by means of correlating the weather observations and market clearing prices. The price-driven methods were tested in the Power TAC simulation framework whereas the others use real-world data. Results show that weather data can successfully reduce the forecasting error up to a certain degree. After all, the paper aims to create generic data-driven forecasting models which can be deployed in autonomous trading agents.

**Keywords:** autonomous agents, electricity market, forecasting, trading.

## 1 Introduction

Governments have started to declare their energy transition policies in order to create a greener energy landscape for the future electricity grid [14]. However, the California crisis (2000 - 2001) showed that such intermediate power actors are the most vulnerable, as they have the burden of a strict financial and power management between customers and generators [13]. Nowadays, electricity price forecasting has become more and more vital, as the increasing renewable capacity is leading the price fluctuations.

Many research efforts have been made so far in the field of energy informatics and economics. One of the most comprehensive research papers is published by [9]. The paper reviews reinforcement learning approaches from the decision-support perspective in smart electricity markets. In this work, retail and wholesale trading problems are
handled separately in a broker-centric environment. In terms of competitive decision-
support, [3] proposes predictive machine learning models for supply-chain manage-
ment using a TAC-SCM scenario. In this work, market regimes and future price distri-
butions are clearly defined, using generic statistical models. Another paper proposes a
Markov Decision Process (MDP) to handle wholesale market activities [4]. Besides this
work, many existing papers have confirmed that MDP is one of the proven ways of
handling time-sequential problems [7,9]. Our most recent publication [10], describes
the basic activities of our winning broker agent (AgentUDE) in Power TAC 2014 Fi-
nals. Note that the methods in the experiments are driven by AgentUDE.

This paper proposes a number of electricity price forecasting methods to have a closer look at the forecasting error problem in renewable driven wholesale markets. At the first stage, we offer a hybrid electricity price forecasting approach, using a number of reinforcement learning methods and MDP, which is a modified version of MDP design, introduced by [4]. We use an exponential smoothing operator along with a belief function which is proposed by [12]. The exponential smoothing itself is known as one of the most popular and responsive forecasting methods, but also the short-sighted one. We integrate the belief function to tune the focus of the trader agent in real-time. Therefore, the trader agent can easily detect short-term fluctuations as well as long-term regime changes. The belief function is the model of the environment and continuously updated, as the market clearing prices (MCPs) are broadcasted to market participants. As a final step, the trader agent solves the MDP at each simulation hour to create its price distributions. We test and compare our model in Power Trading Agent Competition (Power TAC) which is a competitive smart grid simulation platform that uses real weather reports, and competitive broker agents which are developed by different research institutes (see Section 3). We define a quadratic loss function to benchmark the error rate of the models.

At the second stage, we use real market data and weather observations to find real-
world weather and MCP correlations in German electricity markets. Afterwards, we
use that correlation to enhance MCP-driven price. To achieve that, we collected hourly
weather observations of eight German cities. Market prices are hourly sampled at EPEX
Spot so that MCP changes are stored in a database, based on the time distance between
the order hour and the delivery hour (see Section 3.2.1). Then, we formalize the process
to improve MCP-driven prices. The results show that weather observations are signifi-
cantly improving the prediction accuracy.

The structure of the paper is organized as follows. The Power TAC scenario is given in Section 2. Afterwards, Section 3 details the framework model and explains all of the proposed approaches. Finally, the paper is concluded in Section 4 with an overview to our future work.

## 2 Power TAC Scenario

Smart grid simulation platforms have become more and more popular as liberalized electricity markets and decentralized power generation challenge the volatile balance of electricity demand and supply. Simulations aim to address these challenges to create
a vision of sustainable smart grid ecosystems. Power Trading Agent Competition (Power TAC) is one of the most powerful and robust open-source smart grid simulation platforms that brings electricity brokers and smart market concepts together [2,11]. Fig. 1 depicts the high-level structure of Power TAC.

<figure><img src="image_2.png" /><figcaption>Fig. 1. High-level structure of Power TAC scenario.</figcaption></figure>

The platform integrates various smart grid actors such as customer models, a retail market, a wholesale market, a distribution utility, and autonomous electricity brokers within a single distribution area, currently a city. The main actors within Power TAC are now described in more detail:

- Electricity brokers are business entities that trade as intermediaries to attain good results for their own accounts. They try to attract customers by publishing electricity tariffs in the retail market, i.e. tariff market. The so-called distribution utility closely monitors all brokers in order to evaluate their demand and supply behavior. Imbalanced energy is subject to penalties, which may result in a profit loss that is approximately twice as high as the mean wholesale market price. Therefore, brokers have to trade in the wholesale market in order to cover their net demand.

- Customers are small and medium sized consumers and producers such as households, electric vehicles and small firms. They interact with the environment through electricity tariffs. They can buy or sell electricity, subscribing to appropriate tariffs which are defined in power type, time and money domains.

- Generator companies represent the large power generators or consumers. These actors trade in the wholesale market and manage their commitments for the next several hours up to several weeks.

- Distribution utility operates the grid and manages the imbalances in real-time. It is assumed that distribution utility owns the physical infrastructure. It charges brokers for their net distributed energy per kWh, known as distribution fee. It also manages imbalances and charges brokers for their imbalanced energy, called balancing fee.

While Power TAC is available all year-round for all kinds of simulations, its international competition is conducted only once a year. Research institutes are encouraged
to develop and pre-test their own smart energy brokers. A Power TAC tournament consists of a collection of games, grouped in different game sizes, e.g. with three, five and seven players. Game size indicates the number of competing broker agents. In addition to competing teams, a built-in default broker is always included in the games, i.e. it means two brokers and the default broker compete in a three player game. After all games have ended, profits are summed up and normalized on the basis of each individual game size. The broker with the highest aggregated profit is the winner.

<figure><img src="image_3.png" /><figcaption>Fig. 2. Time slot sequence diagram from brokers’ horizon.</figcaption></figure>

A Power TAC game takes up to a random time slot count, starting from one, cf. Fig. 2 for the activities in a time slot. In the paper, we refer to the current time slot $t$ and time distance $\delta$ to future auction hour (see Table 1 to read more about the notation).

1. Brokers receive signals at every time slot $t$, e.g. current cash balance, MCPs ($cp_t, cp_{t+1}, ..., cp_{t+23}$) and published tariffs by all brokers.

2. Brokers ought to submit orders to the wholesale market in order to procure energy amount $E^{t,\delta}(\text{MWh})$ at current time slot $t$, so that its supply and demand match at future time slot $t + \delta$. For this reason, brokers need to predict future demand as accurate as possible to avoid imbalance penalties.

3. At the end of a time slot, a broker’s cash account is updated based on the profit $\sum_i T_i E_i^t - \sum_\delta^{24} cp_{t,\delta} E^{t,\delta}$ where $T_i$ is the tariff price of the net distributed energy $E_i^t$ at time slot $t$, under tariff $i$. The cost of supplying wholesale energy amount $E^{t,\delta}$ at time slot $t$ is denoted as $\sum_\delta^{24} cp_{t,\delta} E^{t,\delta}$. Imbalance penalty $(\sum_i E_i^t - \sum_\delta^{24} E^{t,\delta}) P_{imb}$ is debited from the broker’s cash account, multiplying the net imbalance with the balancing fee of $P_{imb}$ (per unit).

4. In addition to the tariff value, tariff activities like customer sign ups or withdrawals are subject to payment due to bonus and early withdrawal payment parameters of the according tariffs.

5. Brokers pay distribution fee for each energy unit if market power is distributed or local power is traded in the wholesale market.

6. At the end of the time slot, brokers receive information about net distribution and imbalance volumes as well as tariff transactions.
7. Customers initially subscribed to the default tariff. Once brokers join in, they evaluate the existing tariffs based on their energy profile. Due to fact that “set and forget” is a common customer behavior, an inertia factor $I_a = (1 - 2^{-n})I$ drives the motivation of customers. Here, n denotes the time slot after the latest subscription. For more details, [6] includes a comprehensive explanation of the customer model.

Apart from the models mentioned above, the simulation core plays a top-level coordinator role between customers, brokers and the Distribution Utility and provides real-world data, such as weather forecasts.

# 3 Electricity Price Forecasting

In this section, we basically propose two forecasting approaches to make price predictions for future auctions. Firstly, we use historical prices to create future prices. At this stage, we use a number of reinforcement method to assess MCPs. Secondly, weather and MCP correlation will be tackled in the second part, as an option to improve the prediction accuracy of the first step.

<table><caption>**Table 1.** Summary of notation.</caption><thead><tr><th>Symbol</th><th>Definition</th></tr></thead><tbody><tr><td>t</td><td>Current time slot t, i.e., order hour.</td></tr><tr><td>δ</td><td>Time slot proximity. Time slot distance of t to the power delivery hour.</td></tr><tr><td>cp<sub>t,δ</sub></td><td>MCP of the wholesale market ordered at t with δ.</td></tr><tr><td>cp̃<sub>t,δ</sub></td><td>MCP-driven forecasted price at t with δ.</td></tr><tr><td>cp̃<sup>Φ</sup><sub>t,δ</sub></td><td>MCP and weather-driven price at t with δ.</td></tr><tr><td>φ<sub>t</sub></td><td>Weather observation report at t.</td></tr><tr><td>Φ<sub>t,δ</sub></td><td>Weather contribution value to be added on cp̃<sub>t,δ</sub>.</td></tr><tr><td>ρ<sup>δ</sup><sub>wind|temp|cloud</sub></td><td>Correlation coefficient of MCP and regarded weather report for δ.</td></tr></tbody></table>

Table 1 defines the key parameters that are used in the paper. In addition to the notations above, we need to explain a number of terms as well. The time slot proximity refers to time between order hour and delivery hour. For example, bidding at 18:00 for the power delivery at 20:00 means that the proximity is 2.

## 3.1 MCP Driven Electricity Price Forecasting

In this section, we outline the design of our MCP-based forecasting model and benchmark its performance, using different learning rates. Additionally, we compare our wholesale bidding performance with other broker agents in Power TAC environment, using strategic prices which are built on forecasted prices.
Price forecasting is one of the most established area in the time-series analysis. However, due to reasons given in the abstract and introduction of the paper, energy markets are getting closer to a non-stationary position. Daily price spikes, rapidly changing trends require a hybrid forecasting solution.

<figure><img src="image_4.png" /><figcaption>Fig. 3. MCPs of 10:00 AM in Power TAC's continuous wholesale market.</figcaption></figure>

<figure><img src="image_5.png" /><figcaption>Fig. 4. Autocorrelations and partial autocorrelation coefficients, using the signal in Fig. 3. From left to right, we firstly take the autocorrelation and partial autocorrelation of the signal. The second column indicates the correlations after the first-order non-seasonal difference. The third column shows the correlations of the signal after the first-order seasonal difference.</figcaption></figure>

Fig 4. Illustrates a price signal from a Power TAC game. As seen, the signal is stationary and seasonal. Therefore, we can pick a simple seasonal autoregressive integrated moving average (SARIMA) model, analyzing the autocorrelation and partial autocorrelation coefficients in Fig. 3. As noticed, there is a strong seasonality at lag 24 as well as a non-seasonal spike at lag 1. For simplicity, we ignore the moving averages and take SARIMA(1,0,0)x(0,1,0)₂₄ model to describe the forecasting problem. Therefore, the formula can be rewritten as:

$$ \hat{Y}_{t+1} = (Y_t - Y_{t-24}) + Y_{t-23} \quad (1) $$
Where $\hat{Y}_{t+1}$ is the prediction of the next time slot at current time slot $t$ whereas $Y$ values denote historical prices. The problem in the formula is the age of some regression terms such as $Y_{t-23}$ and $Y_{t-24}$. Motivating from the strong correlation in partial autocorrelation of seasonal difference, we replace those aged regression terms with a robust model, using dynamic programming technique so that our forecasting model can avoid price spikes caused by outlier historical prices.

Our forecasting model uses a dynamic programming technique to implement the similar-hour concept [12] in order to make predictions in continuous electricity markets. The similar-hour concept is based on searching the past data for hours with characteristics similar to the predicted hour. Generally speaking, the trader agent has almost same MCPs everyday while bidding at 02:00 for the next 24 hours. In other words, the agent takes similar historical market prices into account, while submitting bids to 03:00, 04:00, ..., 02:00 (next day). We use a belief function as the model of the environment. The belief function is updated continuously as the MCPs are broadcasted to brokers.

We formalize the sequential bidding problem as an MDP, formally described by [7]. Each hour of day (24) is represented by another Markov Process. It means that at each time slot, there are 24 concurrent bidding processes. Each process has 25 states. One of those states is terminal state {completed}. The rest of the states denote the time slot proximity between order hour and delivery hour. Let $P_{14}$ be the process of delivery hour 14:00. Then $P_2$ is in the state 6 and 1 at the order hours 08:00 and 13:00, respectively. Our MDP is defined as follows:

- **States**: $S \in \{1, ..., 24, completed\}$

- **Terminal state**: {completed}

- **Reward**: $R(s', a) = \begin{cases} 1 : s' = \{completed\} \\ 0 : \text{otherwise} \end{cases}$

- **Actions**: $a_s \in \mathbb{Z}$

- **Transitions**: State $s$ transitions to 'completed', if a bid fully clears. Otherwise, it transitions to $s-1$.

Here, action values are limit prices, provided by a value function $V^*(s)$. The value function basically maximizes the sum of expected sum of rewards, and theoretically replaces the term $(Y_{t-23} - Y_{t-24})$, given in Formula 1. The model of the environment is represented by a belief function $f(s, a)$, which is a modified version of a work by [12] and influenced by Q-learning concept [8]. However, Tesauro keeps the probability of a given price by harvesting historical data. In our case, we only keep the weights of changes of two sequential MCP's as the problem defined in Formula 1. Therefore, the belief function $f(s, a)$ points to weights of $a \in \xi_a$, given a state $s$, where higher values mean higher probability of reward occurrence where $\xi_a$ is the set of actions, $\{a \in \mathbb{Z} | -500 \le a \le 500\}$. Since our reward function is a kind of counting process, we are interested in the reward occurrence in the belief function. The action with highest probability ought to result in transition to {completed}.

As time proceeds to $t+1$, the belief functions $f(s, a)$ is updated for $\forall a \in \xi_a$, as MCPs broadcasted to brokers. In brief, MCP's are supervising and reforming the belief function based on the market results. Therefore, the agent does not need to act to learn
and update its model. Following formula updates the belief function, using a learning rate $\alpha$ and a reward function. Note that only MCP's are positively rewarded whereas other actions are rewarded with a zero value (Formula 3). This way, in turn, provides a normalization process on the action-state vector:

$$f_{t+1}(s_t, a_t) = f_t(s_t, a_t) * \alpha + R(s_{t+1}, a_t) * (1 - \alpha) \quad (2)$$

$$s_{t+1} = \begin{cases} 'completed': MCP = a_t \\ s_t - 1: \text{otherwise} \end{cases} \quad (3)$$

Where (1) and (2) are subject to $0 \le \alpha \le 1$.

To solve MDP, we use value iteration method to find the expected sum of rewards. The value function $V^*(s)$ takes a probability density function (pdf), $F_s(a)$ where $\mu$ and $\sigma$ parameters of the normal distribution are obtained from the values of $f(s, a)$, given a state $s$ for $\forall a \in \mathbb{Z}$. Following value function, $V^*(s)$ solves our MDP and creates a bid value, using an exponential smoothing operator. Here, the exponential smoothing operator refers to the non-seasonal auto regression term in Formula 1.

$$V^*(s) = \begin{cases} cp'_{s}: s = 24 \\ cp'_{s+1} + \arg\max_a F_s(s): \text{otherwise} \end{cases} \quad (4)$$

Where exponential smoothing operator is defined as $cp'_{s} = cp_{s}(\beta) + cp'_{s}(1 - \beta)$ and subject to $0 \le \beta \le 1$. Since there is no seasonal difference available at state $s$, we only use an exponential smoothing value.

<table><thead><tr><th colspan="2">Algorithm 1 Non-strategic electricity price forecasting</th></tr></thead><tbody><tr><td>1.</td><td>Needed Energy [1, 2, ..., 24] = computeNeededEnergy()</td></tr><tr><td>2.</td><td>Update f(s,a) and cp'_{s} &larr; auctionResults()</td></tr><tr><td>3.</td><td>For state s in [1,2, ..., 24] do</td></tr><tr><td>4.</td><td>&nbsp;&nbsp;&nbsp;&nbsp;if s = 24 then</td></tr><tr><td>5.</td><td>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;Limit prices [24] = cp'_{s}</td></tr><tr><td>6.</td><td>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;if s &lt; 24 then</td></tr><tr><td>7.</td><td>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;Limit prices [1, 2, ..., 23] = cp'_{s+1} + arg max_a F_s(a)</td></tr><tr><td>8.</td><td>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;Submit bids [1, 2, ..., 24] &larr; Limit prices</td></tr></tbody></table>

The algorithm above summarizes the process. The trader agent collects data from the auction results and update its knowledge. Then, it creates 24 predictions solving the model on hand. As the time proceeds, belief function updates itself. Overall, the system updates itself with reinforcements and combine the benefits of short-term and long-term methods. The reason of using two heterogeneous methods together is to adapt and optimize the trader agent against changing market conditions with little data.
### 3.1.1 Experimental Setup and Results

In the experiments, we use our broker agent AgentUDE15 [10] to benchmark our model (cf., Table 2 for the broker settings). For more details, see the publication which describes the algorithms used in AgentUDE.

We arranged a tournament to create different game variations, and picked well respected and competitive brokers of the recent years: cwiBroker15, CrocodileAgent15, Maxon15 and TacTex14 [1,4,5,15,17]. The suffixes, at the end of broker names indicate the year of release. Due to the number of available brokers, all the games are defined as 3-player to diverse the trading environment. Since we use AgentUDE15 as a testbed application, it is included in all games. Therefore, 3-player game actually means that AgentUDE15 competes with two other brokers as well as a default-broker. All brokers have the same chance of competing with AgentUDE15. The requirements above make 6 game combinations possible. We multiplied the number of games by two and set 12 games in total. We used 1.3.0-Snapshot version of the Power TAC environment and the relevant output was processed in Matlab 2015b.

AgentUDE15 starts a game without offline data, i.e. belief matrix. The belief matrix is filled along the game. We set a symbolic energy procurement amount as 0.1 MWh to make it price taker. The broker has no activity in retail and balancing market.

An absolute error loss function L measures the accuracy of the predictions:

$$L(\tilde{c}\tilde{p}, cp) = \left( \frac{\tilde{c}\tilde{p} - cp}{cp} \right)^2 \qquad (5)$$

Here, the lost function converges to zero, as the output and estimated price get close to each other. Since it is a quadratic function, error values are always positive and the higher error values mean the less prediction accuracy.

<figure><img src="image_6.png" /><figcaption>Fig. 5. Average trading volume by time slot proximity.</figcaption></figure>
<figure><img src="image_7.png" /><figcaption>**Fig. 6.** Error rates of forecasting model, given learning rates, $\alpha=0.2$, $\alpha=0.4$, $\alpha=0.6$ and $\alpha=0.8$. The higher $\alpha$ values mean more conservative behavior (far-sighted).</figcaption></figure>

Fig. 6 summarizes the simulation results from a graphical perspective. Comparing different learning rates, the model seems to be successful at far-sighted mode. This output meets the expectation, found in the figure of partial auto correlation of seasonal difference (see Fig. 4). Therefore, the historical price signal seems partially stationary. However, the proposed method can also be used in non-stationary markets due to exponential smoothing terms in it.

<table><caption>**Table 2.** Average and weighted errors.</caption><thead><tr><th></th><th>α = 0.2</th><th>α = 0.4</th><th>α = 0.6</th><th>α = 0.8</th></tr></thead><tbody><tr><th>Average Loss</th><td>0.0274</td><td>0.0210</td><td>0.0089</td><td>0.0039</td></tr><tr><th>Weighted Loss</th><td>0.0169</td><td>0.0130</td><td>0.0061</td><td>0.0029</td></tr></tbody></table>

Table 2 summarizes the overall performance by learning rate. Here, average loss refers to arithmetic mean of all time slot proximities, whereas weighted loss uses weighted arithmetic mean, considering trading volumes (see Fig. 5). Let $L$ and actual market price be 0.0029 and 25 EUR, respectively. Then 26.34 EUR and 23.66 EUR would be the upper and lower boundaries of the forecasting model.

### 3.1.2 Strategic Bidding and Results

Forecasted prices usually known as truthful information. However, these predictions are not directly submitted to markets by brokers. In order to make the model comparable, forecasted prices must be transformed into strategic prices.

Forecasted prices constitute 24 price distributions where $\mu_{hour}$ and $\sigma_{hour}$ are mean and standard deviation of an hour. We finalize the transformation in two steps:

$$ \text{— Strategic prices } [1, 2, \dots, 24] = [\text{balancingPrice}, \dots, (\mu_{t+24} - \sigma_{t+24})] $$

$$ \text{— Strategic prices } [1, 2, \dots, 24] * = [1 + p_{t+1,\delta=1}, \dots, 1 + p_{t+24,\delta=24}] $$

Where probability of $p_{t,\delta}$ is defined as:
$$p_{t,\delta} = \frac{\sum_{t,s}^{\text{clearingProximity}=\text{proximity}} \text{trading volume}_t}{\sum \text{trading volume}_t} \quad (6)$$

In the first step of the transformation, we assign prices to enabled auctions, starting from the first standard deviation before the mean up to the balancing price. The balancing price is a dynamic variable which is recalculated at every time slot, based on the balancing market reports. Higher proximities are likely to get lower prices. In the second step, we take trading volume into the account. To do that, we scan historical trading volumes, tracking the same bidding proximities. Higher volume probability means higher strategic price for the given proximity.

We repeated the tournament with the same settings as the previous experiment. This time, AgentUDE15 is fully functional on the Power TAC markets, and submits strategic prices instead of forecasted prices. The procurement amount is determined through a demand prediction process, which is out of the scope in the paper.

<table><caption>**Table 3.** Average and weighted errors.</caption><thead><tr><th>Broker</th><th>Avg. Bidding Cost (EUR)</th><th>Weighted Bidding Cost (EUR)</th><th>Avg. Imbalance per TS (kWh)</th></tr></thead><tbody><tr><td>AgentUDE15</td><td>61.04</td><td>52.95</td><td>460</td></tr><tr><td>CrocodileAgent15</td><td>70.32</td><td>67.65</td><td>-170</td></tr><tr><td>cwiBroker15</td><td>55.55</td><td>52.93</td><td>-1738</td></tr><tr><td>Maxon15</td><td>45.93</td><td>43.80</td><td>-3460</td></tr><tr><td>TacTex15</td><td>54.06</td><td>54.29</td><td>-473</td></tr></tbody></table>

Table 3 summarizes the bidding performances. The weighted bidding cost is the realistic indicator which takes clearing volume into account. AgentUDE15 performs on a decent level, having a positive average imbalance per time slot. Besides, there are two extreme bidding schemes in the table: CrocodileAgent15 and Maxon15 follows a generous and stingy bidding policies, respectively. However, those policies eventually result in either high or low imbalance activity. Therefore, brokers have plan their procurements considering the balance of cost and imbalance penalties.

## 3.2 Weather-aided Forecasting

Wholesale energy prices have become correlated with weather conditions as the installed capacity of renewable energy production increases. Germany is a typical example of this phenomenon, time to time selling power to its neighbors at negative prices when it is an extreme windy or sunny day. In this part, we find a correlation between prices and weather conditions.

Due to a number of reasons, weather conditions have different correlation coefficients for different time slot proximities. To consider that, we calculate the correlations for each proximity, using the formula below:
$$ \rho_{wind|temp|cloud}^{\delta} = corr \left( \begin{pmatrix} \varphi_{1+\delta} \\ \varphi_{2+\delta} \\ \vdots \\ \varphi_{n+\delta} \end{pmatrix}, \begin{pmatrix} cp_1^{\delta} \\ cp_2^{\delta} \\ \vdots \\ cp_n^{\delta} \end{pmatrix} \right) \quad (7) $$

Where $corr(a, b) = \frac{Cov(a,b)}{\sigma_a \sigma_b}$, which is known as Pearson product-moment correlation coefficient. Here, weather observations are mapped to the relevant time slot proximity, as weather forecast data is not available for the experimental date. For the correlation of the cloud cover, the formula skips evening hours as the sun only shines at certain hours. Therefore, correlation coefficient of the cloud cover takes only hours between 6:00 and 18:00. This way provides a clear result due to discarding unrelated hours from the correlation vector. Following figure illustrates the correlation coefficient of wind, temperature and cloud cover at a glance.

<figure><img src="image_8.png" /><figcaption>Fig. 7. Correlation coefficient of wind, temperature and cloud cover by time slot proximity.</figcaption></figure>

<figure><img src="image_9.png" /><figcaption>Fig. 8. Historical correlation coefficients of weather reports and average hourly market price.</figcaption></figure>

Fig. 7 shows a snapshot of correlation results whereas Fig. 8 illustrates the average coefficients over the time. Here, positive correlation means that price increase as the regarded weather report value increases. Likewise, negative correlation refers to an inverse proportion. The correlation of the temperature seems seasonal, getting positive value in summer and negative in winter. The cloud cover usually has a positive correlation whereas the wind has a negative value.
The next question is: How to integrate those coefficients to the price-driven predictions. We calculate a contribution parameter, $\Phi_{t,\delta}$ at order time slot $t$ to bid for a future time slot with proximity $\delta$, in order to calculate the value, to be added on forecasted price.

$$ \Phi_{t,\delta}^{wind|temp|cloud} = (\phi_{t+\delta} - \phi_{(t-24)+\delta}) \rho_{wind|temp|cloud} \tilde{cp}_{t+\delta} \quad (8) $$

Where $\phi$ denotes normalized weather observation, which is subject to $0 \le \phi \le 1$. Likewise, $\tilde{cp}$ is a predicted value, for the given future time slot. Contribution amount, $\Phi_{t,\delta}$ can be positive or negative. Note that, $\Phi_{t,\delta}^{cloud}$ is set to zero, in case $t + \delta$ is an evening hour. After all, predicted price $\tilde{cp}_{t+\delta}^{\Phi}$ is derived by adding contribution amount $\Phi_{t,\delta}$ to the cp-driven prediction:

$$ \tilde{cp}_{t+\delta}^{\Phi} = \tilde{cp}_{t,\delta} + \Phi_{t,\delta} \quad (9) $$

### 3.2.1 Experimental Setup and Results

Weather correlation approach was tested using real-world data. Market data was hourly sampled from the EPEX Intraday Continuous Market (DE/AT). The range of time slot proximity is $1 \le \delta \le 24$, which means that we start sampling data, 24 hours prior to the delivery hour. We keep sampling the prices until the gap between current time and delivery time closes. We have 24 concurrent sampling process for the next 24 hours. Thus, our database constitutes a price history for individual auction hours, detailing how the price changes as delivery time approaches. Hourly weather observations were fetched through METAR<fnref n="1" /> systems, which are installed at airports. Observation locations are chosen as Düsseldorf, Hamburg, Berlin, Munich, Stuttgart, Frankfurt, Hannover and Dresden so that observations are weighted equally to find average values for Germany. The experimental data starts from the first day of April until the end of May 2016. The experiment only covers the wind and cloud cover information as the temperature correlation is not statistically satisfactory for the given date interval. Matlab R2015b is used to compute relevant output. For simplicity, the price-driven forecasting model was choosen as the following formula:

$$ \hat{Y}_{t+1} = Y_{t-23} \quad (10) $$

$$ L(\tilde{cp}, cp) = \left( \frac{\tilde{cp} - cp}{cp} \right) \quad (11) $$

<footnote n="1">¹ Meteorological Terminal Air Report is a weather information format that is used by aeronautic purposes.</footnote>
We apply the similar hour approach by taking the MCP of the previous day. Formula 11 depicts the error function to measure the errors. Then, we calculate the error reduction, using the formula below.

$$errorReduction = L_{weather} - L_{price} \qquad (12)$$

Here, we take the difference of the errors as an indicator of the error reduction. Here, $L_{price}$ is solved by Formula10. However, $L_{weather}$ requires an additional term $\Phi_{t+1}$ to be added to $\hat{Y}_{t+1}$. Thus, $L_{weather} - L_{price}$ yields the error reduction for the given time slot and proximity.

<figure><img src="image_10.png" /><figcaption>**Fig. 9.** Reduced forecasting error by proximity. Forecasted price enhanced by wind and cloud cover data.</figcaption></figure>

Fig. 9 illustrates the reduced forecasting error, using the wind and cloud cover data. As seen, forecasting error was reduced up to 0.15. The most significant contribution appears on the first future time slot, as it is the most volatile auction slot in our data set.

# 4 Conclusion and Future Work

In this paper, we proposed data-driven approaches to create a forecasting model. Various reinforcement learning tasks were deployed to create an adaptable and transferable trading mechanism which can be used in similar markets. Secondly, weather reports and forecasts were taken into account to reduce prediction errors. Results show that there is a strong correlation between weather and electricity prices (in Germany), which often drives wholesale market prices to extreme points. Using the method in the paper, price-driven error has been slightly reduced.

Since this is an on-going research, following tasks stand as a future work:

- Uncertainty analysis is needed to have a better understanding of the parameters (e.g., load, supply, reserves, congestion, etc.), influencing the market prices. These parameters may interfere the effects of each other on the market price.

- Strategic bidding has been shortly introduced in the paper. However, the energy amount to be procured needs to be focused deeply. A number of machine learning
methods will be used to identify economic environments and future trends, using
distributed energy storage customers for the trading flexibility.

In this work, we introduce our price forecasting approaches, used in our broker agent.
The controlled experiments show that the proposed forecasting model successfully sup-
ports our strategic bidding, outperforming many of other broker agents in terms of low
cost and low imbalance. Besides, the contribution of weather reports is also introduced
in the paper, which reduces the error rate. The latter is planned to be used in Power
TAC 2016 Finals.

5 References

1. Liefers, Bart, Jasper Hoogland, and Han La Poutré. "A successful broker agent for power TAC." In Agent-Mediated Electronic Commerce. Designing Trading Strategies and Mechanisms for Electronic Markets, pp. 99-113. Springer International Publishing, (2014).

2. Ketter, W., Collins, J., Reddy, P. P., Weerdt, M. D. 2015. The 2015 Power Trading Agent Competition. ERIM Report Series Reference No. ERS-2015-001-LIS.

3. Ketter, W., Collins, J., Gini, M., Gupta, A., Schrater, P. (2009). Detecting and forecasting economic regimes in multi-agent automated exchanges. Decision Support Systems, 47(4), 307-318.

4. Urieli, D., Stone, P., 2014. Tactex'13: a champion adaptive power trading agent. In Proceedings of the 2014 international conference on Autonomous agents and multi-agent systems (pp. 1447-1448). International Foundation for Autonomous Agents and Multiagent Systems.

5. Babic, J. and Podobnik V.: An Analysis of Power Trading Agent Competition 2014. Agent Mediated Electronic Commerce. Volume 187, 2014, 1-15 (2014).

6. Reddy, P.P. and Veloso, M.M., 2012, July. Factored Models for Multiscale Decision-Making in Smart Grid Customers. In AAAI.

7. Sutton, R.S. and Barto, A.G., 1998. Reinforcement learning: An introduction (Vol. 1, No. 1). Cambridge: MIT press.

8. Watkins, Christopher JCH, and Peter Dayan. "Q-learning." Machine learning, no. 3-4 (1992): 279-292.

9. Peters, M., Ketter W., Saar-Tsechansky M., Collins, J., A reinforcement learning approach to autonomous decision-making in smart electricity markets. Machine learning 92, no. 1 (2013): 5-39.

10. Ozdemir S., Unland, R., A Winner Agent in a Smart Grid Simulation Platform. 2015 IEEE/WIC/ACM International Conference on Intelligent Agent Technology (IAT'15).

11. Ketter, W., Collins, J., & Reddy, P. (2013). Power TAC: A competitive economic simulation of the smart grid. Energy Economics, 39, 262-270.

12. Tesauro, G., & Bredin, J. L. (2002, July). Strategic sequential bidding in auctions using dynamic programming. In Proceedings of the first international joint confer-ence on Auton-omous agents and multiagent systems: part 2 (pp. 591-598). ACM.

13. Weron, R. (2014). Electricity price forecasting: A review of the state-of-the-art with a look into the future. International Journal of Forecasting, 30(4), 1030-1081.

14. Morris, C., & Pehnt, M. (2015, July). Energy Transition. The German Ener-giewende. Hein-rich Böll Foundation.