Package: entsoe-client
Disclaimer: This library was written to experiment with various concepts.
The submodulesentsoe_client.ParameterTypesandentsoe_client.Queriesare straight forward.
To refactorentsoe_client.Parsers, the next steps would be to establish fixtures,
standardize flattening of the XML tree and decouple the records from the DataFrame library.ENTSO-E ClientFormulate readable queries and handle data in Pandas,
including an exhaustive set of pre-defined queries.>>>importrequests>>>fromlxmlimportobjectify>>>fromlxml.etreeimportdump>>>url='https://transparency.entsoe.eu/api?'\...'documentType=A81&businessType=A95&psrType=A04&type_MarketAgreement.Type=A01&controlArea_Domain=10YNL----------L'\...f'&periodStart=202101010000&periodEnd=202104010000&securityToken={api_key}'>>>response=requests.Session().get(url=url)>>>element=objectify.fromstring(response.content)>>>dump(element)<Balancing_MarketDocumentxmlns="urn:iec62325.351:tc57wg16:451-6:balancingdocument:3:0"><mRID>051b91beed574b48b4548214e9001afc</mRID><revisionNumber>1</revisionNumber><type>A81</type><process.processType>A34</process.processType><sender_MarketParticipant.mRIDcodingScheme="A01">10X1001A1001A450</sender_MarketParticipant.mRID><sender_MarketParticipant.marketRole.type>A32</sender_MarketParticipant.marketRole.type><receiver_MarketParticipant.mRIDcodingScheme="A01">10X1001A1001A450</receiver_MarketParticipant.mRID><receiver_MarketParticipant.marketRole.type>A33</receiver_MarketParticipant.marketRole.type><createdDateTime>2021-10-04T18:12:43Z</createdDateTime><controlArea_Domain.mRIDcodingScheme="A01">10YNL----------L</controlArea_Domain.mRID><period.timeInterval><start>2020-12-31T23:00Z</start><end>2021-03-31T22:00Z</end></period.timeInterval><TimeSeries><mRID>1</mRID><businessType>A95</businessType><type_MarketAgreement.type>A01</type_MarketAgreement.type><mktPSRType.psrType>A04</mktPSRType.psrType><flowDirection.direction>A03</flowDirection.direction><quantity_Measure_Unit.name>MAW</quantity_Measure_Unit.name><curveType>A01</curveType><Period><timeInterval><start>2020-12-31T23:00Z</start><end>2021-01-01T23:00Z</end></timeInterval><resolution>PT60M</resolution><Point><position>1</position><quantity>44</quantity></Point><Point><position>2</position><quantity>44</quantity>[...]becomes>>>importentsoe_clientasec>>>fromentsoe_client.ParameterTypesimport*>>>client=ec.Client(api_key)>>>parser=ec.Parser>>>query=ec.Query(...documentType=DocumentType("Contracted reserves"),...psrType=PsrType("Generation"),...businessType=BusinessType("Frequency containment reserve"),...controlArea_Domain=Area("NL"),...type_MarketAgreementType=MarketAgreementType("Daily"),...periodStart="2021-01-01T00:00",...periodEnd="2021-04-01T00:00"...)>>>response=client(query)>>>df=parser.parse(response)>>>df.iloc[:,:3].head()positionquantityPeriod.timeInterval.start...2020-12-3123:00:00+00:001442020-12-31T23:00Z2021-01-0100:00:00+00:002442020-12-31T23:00Z2021-01-0101:00:00+00:003442020-12-31T23:00Z2021-01-0102:00:00+00:004442020-12-31T23:00Z2021-01-0103:00:00+00:005442020-12-31T23:00Z...predefined queries are subset of the generic Query class, covering all examples of theENTSO-E API guide.>>>predefined_query=ec.Queries.Balancing.AmountOfBalancingReservesUnderContract(...controlArea_Domain=Area("NL"),...type_MarketAgreementType=MarketAgreementType("Daily"),...psrType=PsrType("Generation"),...periodStart="2021-01-01T00:00",...periodEnd="2021-04-01T00:00"...)...>>>predefined_query()==query()TrueENTSO-E Clientenables straight-forward access toallof the data atENTSO-E Transparency Platform.Query templates abstract the API specifics through Enumerated types.Parse responses into Pandas DataFrames without loss ofanyinformation.The separation of Queries, Client and Parser with their hierarchical abstractions keep the package extensible and maintainable. A pipeline from Query to DataFrame is trivial, preserving the ability to customize steps in between.The implementation relies primarily on theTransparency Platform restful API - user guide.
TheManual of Procedures (MoP)documents provide
further insight on thebusiness requirements specification.
Further information can be found in theElectronic Data Interchange (EDI) Library.Main contributionsExhaustive List of ParameterTypes.These allow mapping between natural language and the codes required
for GET requests, e.g.DocumentType.A85 == DocumentType("Imbalance price").
This feature allows keeping track of queries without jumping between documents or adding comments.Exhaustive List of Pre-defined Queries from ENTSO-E API Guide.ENTSO-E API Guideis a minial set for any API connector to implement and reflects all dashboards on
ENTSO-E Transparency Platform.ParsersResponseDocumentscome in XML schema which can be parsed into pandas DataFrames.Implemented: GL_MarketDocuments, TransmissionNetwork_MarketDocuments,
Publication_MarketDocuments and Balancing_MarketDocuments.Missing: Outages, Congestion Management and System Operations.Nevertheless, ENTSO-E Client seeks to be minimal to go from Query to DataFrame and requires domain-
knowledge on how to formulate queries and interpret various columns of a parsed response.ENTSO-E relies on many codes (Type) to map to desired queries.
Types are encoded in Enum classes with a .help() function to list the all.
They can be addressed through Type[code] or Type(string), making interaction
easy. HTTP requests and responses usually require thecode, whereas we
want to formulate the query as a human-readablestring.from entsoe_client import Queries
from entsoe_client.ParameterTypes import *

Queries.Transmission.CapacityAllocatedOutsideEU(
        out_Domain=Area('SK'),
        in_Domain=Area('UA_BEI'),
        marketAgreementType=MarketAgreementType('Daily'), # Original code: A01
        auctionType=AuctionType('Explicit'), # Original code: A02
        auctionCategory=AuctionCategory('Hourly'), # Original code: A04
        classificationSequence_AttributeInstanceComponent_Position=1,
        periodStart=201601012300,
        periodEnd=201601022300)>>> ParameterTypes.DocumentType['A25'] == ParameterTypes.DocumentType('Allocation result document')
True
>>> ec.ParameterTypes.DocumentType.help()
--- DocumentType ---
API_PARAMETER: DESCRIPTION
[...]
A25: Allocation result document
A71: Generation forecast
A72: Reservoir filling information
A73: Actual generation
A85: Imbalance prices
A86: Imbalance volume
[...]
API_PARAMETER: DESCRIPTION
--- DocumentType ---
>>> ec.ParameterTypes.BusinessType.help()
--- BusinessType ---
API_PARAMETER: DESCRIPTION
[...]
A25: General Capacity Information
A29: Already allocated capacity(AAC)
A97: Manual frequency restoration reserve
B08: Total nominated capacity
C22: Shared Balancing Reserve Capacity
C24: Actual reserve capacity
[...]
API_PARAMETER: DESCRIPTION
--- BusinessType ---#shortened from sample_plot.py
import entsoe_client as ec
from settings import api_key

# Instantiate Client, Parser and Query.
client = ec.Client(api_key)
parser = ec.Parser()
query = ec.Queries.Generation.AggregatedGenerationPerType(
    in_Domain=ec.ParameterTypes.Area('DE_LU'),
    periodStart=202109050200,
    periodEnd=202109070200)

# Extract data.
response = client(query)
df = parser(response)
[...]

# Transform data.
production = df[~consumption_mask][['quantity', 'TimeSeries.MktPSRType.psrType']]
## PsrType, e.g. `B01` := `Biomass`.
production['GenerationType'] = production['TimeSeries.MktPSRType.psrType']. \
    apply(lambda x: ParameterTypes.PsrType[x].value) # Map ENTSO-E PsrTypes into human-readable string.
production_by_type = pd.pivot_table(production,
                                    index=production.index,
                                    columns='GenerationType',
                                    values='quantity')
[...]
# Plot.
production_by_type.plot.bar(title="Production by Generation Type in DE-LU",
                            xlabel="UTC",
                            ylabel='MWh',
                            ax=ax,
                            **plot_params)
[...]