DOI: 10.5281/zenodo.18609860
title: Operational Optimization of PV Assisted Heat Pump Systems Using Reinforcement Learning - Supplemental material

description: This repository provides the supplemental material for the paper Operational Optimization of PV Assisted Heat Pump Systems Using Reinforcement Learning - Supplemental material written by Andreas Heinz, Birgit Hofer, and Franz Wotawa submitted to the 4th International Sustainable Energy Conference, 2026.
description: General notes



System C in the supplementatal material is system B in the paper.

Even though the weather data files are named 2014-2024, only the first nine years are used during the training phase.

The scenarios in the paper have different names in the supplemental material:





Scenario
Supplemental material




1
 


2a
tinyEnv_


2b
tinyEnv2_


3a
single_action_


3b
single_action_tinyEnv_


3c
single_action_tinyEnv2_




Folder structure



results_raw contains the actions and environment information that was created during conducting the experiments.



Notation for training files: {algorithm}_{years}Y_{steps_per_day}SPD_NCOE_HP_{system}_{wheater_dataset}_{TES_size}_{tarriff}_{scenario}__training.csv, e.g. PPO_40Y_24SPD__NCOE_HP__HD_C_2014-2024_23_const_tinyEnv2__training.csv contains the data produced during 40 years of training with 24 steps per day (i.e. hourly actions) using the PPO algorithm on system C with a TES of 2.3 m3 with a fixed tariff (const) based on the weather data from 2014-2024 (where only the first 9 years are used).

Notation for evaluation files: {name_of_the_trained_model}{system_evaluated}_{wheater_dataset}_{TES_size}_{tariff}_evaluation.csv, e.g. PPO_40Y_24SPD__NCOE_HP__HD_C_2014-2024_23_constA_2023_2_const_evaluation.csv contains the data that was produced when evaluating an agent trained on system C with a TES of 2.3 m3 with a fixed tariff on system A with a TES of 2 m3 using the wheater data from 2023 and a fixed tariff.



results_summary contains for each scenario a spreadsheet summarizing the results of the scenario (average overheating temperatures and NCOE for the year 2023 for each system and TES size).


Structure of the results_raw files



Each row represents the actions set and the environment information obtained. In our case a row contains hourly information.

The meaning of the columns are as follows. Please note that the agent receives only the state information that is indicated in the scenario although the raw files contain more information.





Variable
Description




dt_overch_WW
Action: Setpoint for storage overheating DHW


dt_overch_Heiz
Action: Setpoint for storage overheating space heating


hour_sin
Sinus of the current time


hour_cos
Cosine of the current time


T_sens_WW
Storage temp. – domestic hot water (DHW) zone


T_sens_SH_o
Storage temp. – space heating zone (upper)


T_sens_SH_u
Storage temp. – space heating zone (lower)


W_el_Bezug
Elec. energy drawn from the grid during the last time interval


W_el_Einsp
Elec. energy fed into the grid during the last time interval


Q_HK1
Forecasted heating demand for the current hour


Q_HK_sum24
Total forecasted heating demand over the next 24 hours


W_PV1
Forecasted PV yield for the current hour


W_PV_sum24
Total forecasted PV yield over the next 24 hours


Tamb1
Forecasted outdoor temperature for the current hour


Tamb_avg24
Average forecasted outdoor temperature over the next 24 hours


RelHum1
Forecasted relative humidity for the current hour


P_el_PV_Ues
Current PV excess yield (average over the last 7 minutes)


c_feedin
Electricity feedin tariff


W_PV_sum_last
Sum of PV-yield of the last time intervall


Q_HK_sum_last
Sum of heating energy demand of the last time intervall


NCOE
Net cost of electricity (total)


NCOE_HP
Net cost of electricity (of the heat pump only)


reward
Reward = Net cost of electricity (total)




Structure of results_summary files



avg temp. ww represents the setpoint for storage overheating DHW averaged over the year 2023.

avg temp. heat represents the setpoint for storage overheating space heating averaged over the year 2023.

no overheating and optimal overheating represent the corresponding values of the baseline solutions.

Each row represents a system and TES size that the agent was trained on.

Each column represents a system and TES size used for the evaluation.

The diagonal entries are highlighted in bold and represent those cases where the agent is evaluated on the same system as it was trained on (but with different wheater data).


Acknowledgements

The research described here has been carried out within the FFG AI for Green program as part of the project FO999899931 GreenHeat – The next generation of AI-based services for heat pumps funded by the Federal Ministry for Climate Action, Environment, Energy, Mobility, Innovation and Technology (BMK).