# Wholesale Power to Hydrogen: Adaptive Trading Approaches in a Smart Grid Ecosystem

Serkan Özdemir, Rainer Unland

► To cite this version:

Serkan Özdemir, Rainer Unland. Wholesale Power to Hydrogen: Adaptive Trading Approaches in a Smart Grid Ecosystem. 4th IFIP International Conference on Artificial Intelligence in Theory and Practice (AI 2015), Oct 2015, Daejeon, South Korea. pp.75-82, 10.1007/978-3-319-25261-2_7. hal-01383947

HAL Id: hal-01383947

https://inria.hal.science/hal-01383947

Submitted on 19 Oct 2016

HAL is a multi-disciplinary open access
archive for the deposit and dissemination of sci-
entific research documents, whether they are pub-
lished or not. The documents may come from
teaching and research institutions in France or
abroad, or from public or private research centers.

L'archive ouverte pluridisciplinaire HAL, est
destinée au dépôt et à la diffusion de documents
scientifiques de niveau recherche, publiés ou non,
émanant des établissements d'enseignement et de
recherche français ou étrangers, des laboratoires
publics ou privés.

<figure><img src="image_1.png" /></figure>

Distributed under a Creative Commons Attribution 4.0 International License
# Wholesale Power to Hydrogen: Adaptive Trading Approaches in a Smart Grid Ecosystem

Serkan Özdemir and Rainer Unland

DAWIS, University of Duisburg-Essen, Schützenbahn 70, 45127 Essen
{serkan.oezdemir,rainer.unland}@icb.uni-due.de

**Abstract.** Fossil based liquid fuels, primarily used in transportation systems, are likely to be replaced with renewable resources thanks to energy transition policies. However, shifting from stable energy production (using coal, natural gas) to highly volatile renewable production will bring a number of problems as well. On the other side, tremendous developments in solar and wind power technologies encourage energy investors to maximize their contributions over the electricity grid. This highly volatile energy resources bring a strong research question to the attention: How to benefit from excess energy? Power-to-gas seems to be a strong candidate to store excess energy. Besides, power-to-hydrogen is seen as a liquid fuel for fuel cell vehicles. This paper aims to analyze trading approaches of a power-to-hydrogen system to minimize the energy costs. To achieve that, Markov Decision Process (MDP) along with Q-learning is modelled as well as a number of trading approaches. This research aims to reveal the feasibility of hydrogen as a fuel option in future smart grid.

**Keywords:** trading, hydrogen, transportation, simulation, power to gas.

## 1 Introduction

Due to energy transition policies of governments and recent developments in renewable energy technologies, fossil and nuclear based power plants tend to be replaced with renewable resources. Recent developments show that the number of installed capacity will dramatically increase in the near future. Solar siding and roof-top panel technologies are rapidly growing since they have a large footprint compared to other renewable resources. This work assumes that energy transition will shift towards renewables as already planned by many countries.

In case of high renewable penetration, future smart grid will face with a number of challenges, such as meeting the supply and demand in balance. Since the renewable energy production is highly weather-depended, a distributed energy storage is needed at off-peak hours or days to benefit from excess energy. Among other storage options, power-to-gas has the most storage capacity over other technologies [1][9][15]. The first step product of power-to-gas, obtained through electrolysis process, can be used as fuel in fuel cell vehicles. This way is more efficient than methane in terms of energy loss. Hydrogen is also nature friendly fuel and the output of fuel cell vehicles is only water. However hydrogen cannot be delivered to far away due to high pressure
problems. For this reason, on-site production is one of the proposed methods for hydrogen production [8][9][11].

Fuel cells are usually considered as a competitor of battery electric vehicles. However, fuel cell vehicles are replacement of traditional combustion engine vehicles. Moreover, fuel cells are also battery electric vehicles. In addition to all functionalities of battery electric vehicles, fuel cells have high pressure hydrogen tank and fuel cell stack which converts hydrogen into electricity. For this reason, fuel cells can be solution to charging, efficiency and driving-range problems of battery electric vehicles.

This works aims to analyze hydrogen production through electrolysis process on the city level. Power Trading Agent Competition (Power TAC) is selected to simulate future smart grid conditions [16]. A hydrogen station is designed as a server module in which a number of fuel cell and conventional vehicles are simulated. Refilling station consists of an electrolysis unit, high pressure storage unit, dispensers and on-site renewable resources. The station is an active participant of a local wholesale market. The wholesale market is a typical hour-ahead market which allows participants to submit orders 24 hours prior to delivery. Proposed Markov Decision model and trading approaches are explained in Section 3.

## 2 State of the Art

Both power-to-hydrogen ($PtH_2$) and fuel cells are quite old concepts. Basically, the electrolysis extracts water into oxygen and hydrogen ($H_2O \rightarrow H_2 + O$). Among different electrolysis approaches, alkaline electrolysis is the most common one in use. Conversion efficiency rate depends on the load, but the typical rate is 60-70% at full load. Hydrogen can be also injected into natural gas grid [14]. In a typical $PtH_2$ power plant, investment and operation-management costs have severe roles on the profitability of the plant. Following table shows the basic inputs and outputs of a $PtH_2$ power plant.

<table><caption>**Table 1.** Inputs and Outputs of a $PtH_2$ power plant and fuel station.</caption><thead><tr><th></th><th>Today</th><th>2030</th></tr></thead><tbody><tr><td>Investment (electrolysis) (IC<sub>e</sub>) [8]</td><td>1750 EUR / kW<sub>input</sub></td><td>700 EUR / kW<sub>input</sub></td></tr><tr><td>Investment (refueling st. + storage) (IC<sub>rs</sub>) [3]</td><td>16 % of IC<sub>e</sub></td><td>8 % of IC<sub>e</sub></td></tr><tr><td>Operational costs [10]</td><td colspan="2">3% of IC<sub>e</sub> + IC<sub>rs</sub></td></tr><tr><td>Water consumption</td><td colspan="2">0.2 l/kW<sub>input</sub></td></tr><tr><td>Hydrogen production</td><td>1 kg / 48 kW</td><td>-0.492</td></tr><tr><td>Oxygen output</td><td>6 kg / 48 kW</td><td>-1.281</td></tr><tr><td>Useful heat</td><td colspan="2">11 % of input.</td></tr><tr><td>Wholesale market fees</td><td colspan="2">15000-25000 EUR/year [7,8]</td></tr><tr><td>Recurring market and grid fees, taxes</td><td>0.1-0.2 EUR/MWh</td><td>Possible incentives.</td></tr></tbody></table>

Besides the advantages of $CO_2$ emission level and driving range, fuel cells also performs a promising well-to-wheel performance for the future transportation. Fol-
lowing table compares roughly the well-to-wheel performances of fuel cell vehicles, battery electric vehicles and diesel vehicles.

<table><caption>**Table 2.** Comparison of different fuel types. Reference vehicles are B segment economy cars of Mercedes, Toyota and Hyundai.</caption><thead><tr><th></th><th>Fuel Cell Vehicle</th><th>Battery Electric vehicle</th><th>Diesel Vehicle</th></tr></thead><tbody><tr><td>Range (100 km)</td><td>1 kg H<sub>2</sub> (48 kWh el. input)</td><td>18 kWh</td><td>5 liter</td></tr><tr><td>Well-to-wheel wholesale<sup>1</sup></td><td>1.2 EUR</td><td>0.62 EUR</td><td>2.9 EUR [2]</td></tr><tr><td>Well-to-wheel retail</td><td>9 EUR [4]</td><td>4.68 EUR [5]</td><td>5.75 EUR [6]</td></tr></tbody></table>

There are a number of possible incentives that are subject to PtH₂ plants. However, on the legal side, some are not matured due to uncertainties on the future fuel options. But the good news is, there are many ongoing acts regarding to hydrogen fuel utilizing renewable electricity. Currently, many companies, such as OMV, Hydrogenics, Toyota and e-on are active in the hydrogen business by producing fuel cell cars, power plants and refilling stations.

On the other side, energy markets have the vital role on power-to-gas power plants and will be more important in the future due to high fluctuations. In the current situation, electric vehicles are exposed to retailer prices since it is not possible for each electric vehicle to trade in wholesale markets. Unlike electric vehicles, power-to-gas power plants and their refilling stations are able to trade in energy markets. For this reason, fuel cell transportation is seen as one of the strongest candidates for the future transportation system.

## 3 Methodology

In order to analyze hydrogen as a fuel option, a Power TAC server module is created. This module simulates a hydrogen refilling station and on-site hydrogen production. Local wholesale market and on-site renewables are electricity resources of hydrogen production. Note that Power TAC is simulated on the city level with a population of about 50 thousand residents, which fits to on-site PtH₂ power plant scenario since the long-haul distribution of hydrogen is not possible [14]. Figure 1 illustrates the schematic landscape of proposed environment.

<footnote n="1">¹ The row indicates wholesale costs without taxes, profits and service fees.</footnote>
<figure><img src="image_2.png" /><figcaption>**Fig. 1.** Simulation model in Power TAC scenario.</figcaption></figure>

The Power Trading Agent Competition (Power TAC) is an open source smart grid simulation platform which consists of wholesale market, tariff market, distribution utility and a number of costumer and producer models. Besides, autonomous brokers are allowed to trade remotely in these markets. The wholesale market is a typical hour-ahead market where the large generator companies, renewable production farms and brokers place their bids and asks for the future time slots. Trading is enabled at future time slots $t+1, ..., t+24$ at current time slot $t$. The retail market allows brokers to build their customer portfolio by means of offering multiple tariffs to local producers and consumers. In the middle of the retail and wholesale markets, the distribution utility keeps track of supply and demand, and charges brokers for their imbalanced energy. Customers are simulated as independent consumer and producer models such as electric vehicles, house-holds, storage units and solar panels. The interaction between customers and brokers takes place in the retail market through tariff subscriptions.
<table><caption>**Table 2.** Parameters of the proposed ecosystem.</caption><thead><tr><th></th><th>Symbol</th><th>Description</th><th>Value</th></tr></thead><tbody><tr><td>Size of electrolysis unit</td><td>S<sup>elec</sup></td><td>Size of unit which converts electricity to H<sub>2</sub>.</td><td>10 MW</td></tr><tr><td>Size of H<sub>2</sub> storage unit</td><td>S<sup>str</sup></td><td>Size of storage unit in which produced H<sub>2</sub> is stored.</td><td>1000 kg</td></tr><tr><td>Electrolyser efficiency vector</td><td>E<sup>elec</sup></td><td>Efficiency vector of an electrolysis unit based on the rate of (electricity input/ S<sup>elec</sup>).</td><td>[0 – 85 %]</td></tr><tr><td>Energy equiv. of 1 MWh electricity</td><td>R<sup>dig</sup></td><td>Equivalence of 1 MWh electrical power to H<sub>2</sub> regardless of losses.</td><td>30 kg H<sub>2</sub></td></tr><tr><td>Learning alpha</td><td>Q<sub>ALPHA</sub></td><td>Alpha parameter of Q-learning.</td><td>0.5</td></tr><tr><td>Learning gamma</td><td>Q<sub>GAMMA</sub></td><td>Gamma parameter of Q-learning</td><td>0.0</td></tr><tr><td>Time horizon</td><td>H</td><td>Total length of a game in time slot.</td><td>8765</td></tr><tr><td>Future time slot</td><td>T</td><td>Future time slots which are enabled for trading.</td><td>24</td></tr><tr><td>Bidding margin</td><td>M</td><td>Margin that is added to the price obtain through MDP.</td><td>5 EUR</td></tr></tbody></table>

The proposed refilling station simulates the following components:

- Electrolysis Unit: Converts input power to hydrogen having an efficiency rate which depends on the size of electrolysis unit and input power.

- Hydrogen Storage Unit: This unit stores the produced hydrogen through electrolysis process. It also supplies hydrogen to dispensers.

- Dispensers: Final end-point where refilling hydrogen to simulated fuel cell vehicles takes place.

- A number of fuel cell and traditional vehicles.

- On-site solar panels and wind mills which supply electricity to electrolysis unit.

- Trader module: Trades in the wholesale market and optimize the costs considering various variables and on-site production.

Trading in the wholesale market is the most significant part of the research since the motivation of the research is to benefit from the excess energy. Unlike electricity retailers, hydrogen trader unit can make flexible decisions and watch cheap prices at future hours thanks to its hydrogen storage unit. Storage unit can easily tolerate several time slots to let trader unit find cheaper energy in an hour-ahead market.

## 3.1 Trading Model

Unlike broker models in the ecosystem, power-to-gas models are supposed to have an appropriate trading model. Brokers have to deal with balancing problems and make careful decisions to avoid balancing penalties. In order to meet supply and demand, brokers might submit extremely generous bids. However, this landscape is a bit different for power-to-gas plants. Thanks to their storage units, they are able to make more flexible decisions. Following statement presents the optimization problem of overall cost over time horizon H.
$$ \min \sum_{t=1}^{H} \sum_{n=1}^{T} e_{n}^{t} \cdot p_{n}^{t} \qquad (1) $$

Where e is energy and p is energy price over time horizon H and enable future time slots T. In order to optimize the wholesale market cost over a time horizon, we propose a trading model using Markov Decision Process (MDP) design, deploying a Q-learning method [17]. In this approach, all hours are represented as 24 individual processes. Each process has 25 states which represent the future time slot proximity as well as completed state. Proposed MDP is designed as follows:

- **States**: $s \in \{1, ..., 24, \text{completed}\}$

- **Terminal state**: $\{\text{completed}\}$

- **Reward function**: In state $\{\text{completed}\}$, the reward function returns 0. Otherwise, it returns 0.

- **Actions**: $\text{limitPrice}_s \in \mathbb{Z}: s = 24$, $\text{limitPrice}_s - \text{clearingPrice}_{s+1} \in \mathbb{Z}: s < 24$.

- **Transition function**: In a state $s \in \{1, ..., 24\}$, if an order fully clears, it transitions to completed state. Otherwise, s transitions to s-1. In every time slot, a new episode starts for processes where $s = 0$.

In this MDP design, actions are defined as difference of limit price and clearing price which cleared in previous state. In other words, actions indicate increment values as state s transitions to s - 1. This way provides many opportunities such as catching price trends regardless of weather conditions. Even if a bid does not clear at $s = 24$, next bids eventually become more adaptive.

In the learning mode, delayed market data is used to update Q-matrix. Therefore, actions are defined as $\text{clearingPrice}_s \in \mathbb{Z}: s = 24$, $\text{clearingPrice}_s - \text{clearingPrice}_{s+1} \in \mathbb{Z}: s < 24$. Following Q-learning formula evaluates the market experience of the agent.

$$ Q_{t+1}(s_t, a_t) = Q_t(s_t, a_t) * \alpha + (R_{t+1} + \gamma * \max_a Q_t(s_{t+1}, a_t)) * (1 - \alpha) \quad (2) $$

Subject to $0 \le \alpha \le 1$ and $0 \le \gamma \le 1$. After each iteration, Q matrix is normalized to make sure that all values are set between 1 and 0.

Solving MDP is straight forward since it only requires searching the right value in the state-action vector. Following formula fetches the predicted price at current time slot t and future time slot T.

$$ MDP(t, T) = \text{clearingPrice}_{t-1, T} + (\arg \max_a Q_t(T - t, a_t)) \quad (3) $$

However, various trading approaches are required to optimize the overall cost. Using predicted prices through MDP's might be helpful to find the future prices. But another procedure is needed to catch cheap prices and skip expensive wholesale prices. To achieve this, we propose 4 trading approaches defined as follows.

- **Stingy**: This bidding approach always submit lower prices than solved prices through MDP. Submitted price will be $\text{solveMDP}(t, T) - M$ where M is set to 5.

- **Neutral**: Solved price through MDP is directly submitted in the order.

- **Generous**: Unlike Stingy approach, it always submit generous prices than solved prices through MDP. Final price will be $\text{solveMDP}(t, T) + M$ where M is set to 5.
- Combined: This strategy combines all of the approaches above. For a particular current time slot *t*, predicted prices (for future time slot *T*) at *t*, *t+1*, ..., *T-1* are divided into 3 price zones which are defined as cheap, normal and expensive. In cheap zone, generous bidding is used to increase the chance of buying cheap energy. In normal price zone, neutral bidding strategy is used. Likewise, stingy bidding is used in expensive price zone to decrease the probability of buying expensive energy.

As a future work, these strategies will be integrated to hydrogen storage optimization problem to minimize overall cost of refilling station.

## 4 Future Work

Proposed design above enables various studies from different perspectives. First, a legal landscape (taxation, incentives and transition policies) and a future projection will be covered under this research. Second, renewables have to be taken into account deeply since they are the main drivers of the future energy production. Within this scope, all of penetration levels will be simulated with the proposed trading methods. Third, size of refilling station, such size of electrolysis unit, storage unit, pumps and so on, will be subject to an optimization problem considering investment and operational costs.

This work aims to control a number of variables.

- Size of electrolysis unit (MW). An electrolysis unit is the most efficient at 25% electricity input.

- Size of hydrogen storage (kg). A bigger size of storage unit can put trader unit into a more flexible position. Therefore, operating modes will be defined based on the hydrogen storage.

- Share of vehicle groups. Percentages of fuel cells, battery electric vehicles and traditional vehicles among all passenger vehicles (cars, buses, vans).

- Production volume (including local producers). Various production rates will be subject to experiments.

- Distribution fee. This fee is paid if the bought energy is originated from wholesale market. Incentives can waive this fee.

- Trading approaches. These approaches will be defined in Markov processes later on.

- Number of retailer/broker companies.

All of these variables are controlled to optimize the cost and investment problems as well as further possible analysis. Existing works in the literature are usually based on the static data or estimations. Power-to-gas is usually considered as a profitability problem or balancing approach which are far away from transportation perspective.
# 5 Conclusion

In summary, the smart grid will bring a lot of benefits such as excess energy. On the other side, fuel cells have all the functionalities of a battery electric vehicle in addition to hydrogen storage and fuel cell stack. This capability provides opportunity to drive with hydrogen or electricity no matter which one is available in the vehicle. Obviously, both ways are nature friendly and do not replace each other.

Proposed trading approaches and design show that power-to-hydrogen units are able to trade in the wholesale markets, making more flexible decisions thank to hydrogen storage units. It is noteworthy that unstable energy production by renewables eventually results in unstable price regime. This highly volatile environment will be the driver of power-to-gas plants.

# 6 References

1. Hall P., Bain P. 2008. Energy-storage technologies and electricity generation. Elsevier.

2. Mineralölwirtschaftsverbande. V. Statistiken–Preise. http://www.mwv.de/index.php/daten/statistikenpreise/?loc=1. (03.12.2014).

3. Pure Energy Centre. Hydrogen refueling station. http://pureenergycentre.com/hydrogen-fueling-station/. (05.12.2014).

4. Kurier. Erste Wasserstoff-Tankstelle: Künftig tanken wir Kilos. http://kurier.at/wirtschaft/1-wasserstoff-tankstelle-kuenftig-tanken-wir-kilos/824.355. (12.12.2014).

5. Verivox. Direktvergleich. http://www.verivox.de/strompreisvergleich. (24.11.2014).

6. Clever Tanken. Aktuelle Diesel, Benzinpreise. http://www.clever-tanken.de. (11.12.2014).

7. Nord Pool Spot. Nordic and Baltic Trading Fees. http://www.nordpoolspot.com/TAS/Fees/Nordic-Baltic/. (11.12.2014).

8. EPEX Spot. Price List. http://static.epexspot.com/document/29089/EPEXSPOT_Price_List_January_2015.pdf. (14.12.2014).

9. Federal Ministry of Transport and Digital Infrastructure (BMVI). 2014. Power-to-Gas (PtG) in transport: Status quo and perspectives for development. Berlin.

10. National Renewable Energy Laboratory. 2014. Hydrogen Station Compression, Storage, and Dispensing Technical Status and Costs.

11. Zero Regio. 2010. The future cost and competitiveness of hydrogen as a transport fuel in Europe.

12. Lizbeth, C. G. M. 2013. Assessment of usage of hydrogen as alternative fuel into NETPLAN (PhD dissertation). Iowa State University.

13. Fuel Cell Today. 2013. Water Electrolysis & Renewable Energy Systems.

14. Sterner, M. 2013. Power-to-Gas: Perspektiven einer jungen Technologie.

15. Reichert, F., Brian, V. M. 2012. Wind-to-Gas-to-Money? Economics and Perspectives of the Power-to-Gas Technology (master thesis). Aalborg University.

16. Ketter, W., Collins, J., Reddy, P. P., Weerdt, M. D. 2015. The 2015 Power Trading Agent Competition. ERIM Report Series Reference No. ERS-2015-001-LIS.

17. Watkins, C. J., Dayan, P. 1992. Q-learning. Machine learning, 8(3-4), 279-292.