DOI: 10.5281/zenodo.3744121
title: Pre-Processed Power Grid Frequency Time Series

description: <strong>Overview</strong><br> This repository contains ready-to-use frequency time series as well as the corresponding pre-processing scripts in python. The data covers three synchronous areas of the European power grid: Continental Europe Great Britain Nordic This work is part of the paper "Predictability of Power Grid Frequency"[1]. Please cite this paper, when using the data and the code. For a detailed documentation of the pre-processing procedure we refer to the supplementary material of the paper. <strong>Data sources</strong><br> We downloaded the frequency recordings from publically available repositories of three different Transmission System Operators (TSOs). <strong>Continental Europe </strong>[2]: We downloaded the data from the German TSO <em>TransnetBW GmbH,</em> which retains the Copyright on the data, but allows to re-publish it upon request [3]. <strong>Great Britain </strong>[4]: The download was supported by National Grid ESO Open Data, which belongs to the British TSO <em>National Grid</em>. They publish the frequency recordings under the NGESO Open License [5]. <strong>Nordic</strong> [6]: We obtained the data from the Finish TSO <em>Fingrid</em>, which provides the data under the open license CC-BY 4.0 [7]. <strong>Content of the repository</strong> <strong>A) Scripts</strong> In the "Download_scripts" folder you will find three scripts to automatically download frequency data from the TSO's websites. In "convert_data_format.py" we save the data with corrected timestamp formats. Missing data is marked as NaN (processing step (1) in the supplementary material of [1]). In "clean_corrupted_data.py" we load the converted data and identify corrupted recordings. We mark them as NaN and clean some of the resulting data holes (processing step (2) in the supplementary material of [1]). The python scripts run with Python 3.7 and with the packages found in "requirements.txt". <strong>B) Data_converted and Data_cleansed</strong><br> The folder "Data_converted" contains the output of "convert_data_format.py" and "Data_cleansed" contains the output of "clean_corrupted_data.py". <strong>File type</strong>: The files are zipped csv-files, where each file comprises one year. <strong>Data format</strong>: The files contain two columns. The first one represents the time stamps in the format <em>Year-Month-Day Hour-Minute-Second</em>, which is given as naive local time. The second column contains the frequency values in Hz. <strong>NaN representation</strong>: We mark corrupted and missing data as "NaN" in the csv-files. <strong>Use cases</strong><br> We point out that this repository can be used in two different was: <strong>Use pre-processed data</strong>: You can directly use the converted or the cleansed data. Note however that both data sets include segments of NaN-values due to missing and corrupted recordings. Only a very small part of the NaN-values were eliminated in the cleansed data to not manipulate the data too much. If your application cannot deal with NaNs, you could build upon the following commands to select the longest interval of valid data from the cleansed data: <pre><code class="language-python">from helper_functions import * import pandas as pd cleansed_data = pd.read_csv('/Path_to_cleansed_data/data.zip', index_col=0, header=None, squeeze=True, parse_dates=[0]) valid_bounds, valid_sizes = true_intervals(~cleansed_data.isnull()) start,end= valid_bounds[ np.argmax(valid_sizes) ] data_without_nan = cleansed_data.iloc[start:end]</code></pre> <strong>Produce your own cleansed data</strong>: Depending on your application, you might want to cleanse the data in a custom way. You can easily add your custom cleansing procedure in "clean_corrupted_data.py" and then produce cleansed data from the raw data in "Data_converted". <strong>License</strong><br> We release the code in the folder "Scripts" under the MIT license [8]. In the case of Nationalgrid and Fingrid, we further release the pre-processed data in the folder "Data_converted" and "Data_cleansed" under the CC-BY 4.0 license [7]. TransnetBW originally did not publish their data under an open license. We have explicitly received the permission to publish the pre-processed version from TransnetBW. However, we cannot publish our pre-processed version under an open license due to the missing license of the original TransnetBW data.
description: We thank Mark Thiele for fruitful discussions. Furthermore, we gratefully acknowledge support from the German Federal Ministry of Education and Research (BMBF grant no. 03EK3055B) and the Helmholtz Association (via the the "Helmholtz School for Data Science in Life, Earth and Energy" (HDS-LEE), the joint initiative "Energy System 2050 - A Contribution of the Research Field Energy" and via the grant No. VH-NG-1025). This project has received funding from the European Union's Horizon 2020 research and innovation programme under the Marie Skłodowska-Curie grant agreement No. 840825.
description: {"references": ["[1] J. Kruse, B. Sch\u00e4fer, and D. Witthaut, \"Predictability of Power Grid Frequency,\" arXiv:2004.09259 [physics, stat], Apr. 2020.", "[2] TransnetBW GmbH. \"Netzfrequenz\". https://www.transnetbw.de/de/strommarkt/systemdienstleistungen/regelenergie-bedarf-und-abruf (accessed on 26.03.2020)", "[3] TransnetBW GmbH. \"Conditions of Use\". https://www.transnetbw.com/en/conditions-of-use (accessed on 26.03.2020)", "[4] National Grid ESO. \"Historic frequency data\". https://www.nationalgrideso.com/balancing-services/frequency-response-services/historic-frequency-data (accessed on 26.03.2020)", "[5] National Grid ESO. \"NGESO Open Licence\". https://national-grid.ckan.io/licence (accessed on 26.03.2020)", "[6] Fingrid Oyj. \"Frequency - historical data\". https://data.fingrid.fi/en/dataset/frequency-historical-data (accessed on 26.03.2020)", "[7] Creative Commons. \"Attribution 4.0 International (CC BY 4.0)\". https://creativecommons.org/licenses/by/4.0/deed.en (accessed on 26.03.2020)", "[8] \"MIT License\". https://choosealicense.com/licenses/mit/ (accessed on 26.03.2020)"]}