# Advancing climate services for the European renewable energy sector through capacity building and user engagement

C.M. Goodess, A. Troccoli, C. Acton, J.A. Añel, P.E. Bett, D.J. Brayshaw, M. de Felice, S.R. Dorling, L. Dubus, L. Penny, et al.

<figure><img src="image_1.png" /></figure>

## To cite this version:

C.M. Goodess, A. Troccoli, C. Acton, J.A. Añel, P.E. Bett, et al. Advancing climate services for the European renewable energy sector through capacity building and user engagement. Climate services, 2019, 16, pp.100139. 10.1016/j.cliser.2019.100139. hal-02402136

HAL Id: hal-02402136

https://minesparis-psl.hal.science/hal-02402136

Submitted on 11 Dec 2019

HAL is a multi-disciplinary open access
archive for the deposit and dissemination of sci-
entific research documents, whether they are pub-
lished or not. The documents may come from
teaching and research institutions in France or
abroad, or from public or private research centers.

L'archive ouverte pluridisciplinaire HAL, est
destinée au dépôt et à la diffusion de documents
scientifiques de niveau recherche, publiés ou non,
émanant des établissements d'enseignement et de
recherche français ou étrangers, des laboratoires
publics ou privés.

<figure><img src="image_2.png" /></figure>

Distributed under a Creative Commons Attribution 4.0 International License
<figure><img src="image_3.png" /></figure>

Original research article

# Advancing climate services for the European renewable energy sector through capacity building and user engagement

<figure><img src="image_4.png" /></figure>

C.M. Goodess<sup>a,*</sup>, A. Troccoli<sup>a,b</sup>, C. Acton<sup>c</sup>, J.A. Añel<sup>d</sup>, P.E. Bett<sup>c</sup>, D.J. Brayshaw<sup>e,f</sup>, M. De Felice<sup>g</sup>, S.R. Dorling<sup>a</sup>, L. Dubus<sup>h</sup>, L. Penny<sup>a</sup>, B. Percy<sup>j</sup>, T. Ranchin<sup>j</sup>, C. Thomas<sup>k</sup>, M. Trolliet<sup>j</sup>, L. Wald<sup>i</sup>

## ARTICLE INFO

### Keywords:

Climate services

Sectoral information system

Renewable energy

User needs and requirements

Stakeholder engagement

Co-production

## ABSTRACT

The development of successful climate services faces a number of challenges, including the identification of the target audience and their needs and requirements, and the effective communication of complex climate information, through engagement with a range of stakeholders. This paper describes how these challenges were tackled during the European Climatic Energy Mixes (ECEM) project, part of the Copernicus Climate Change Service (C3S), in order to deliver a pre-operational, proof-of-concept climate service for the European renewable energy sector. The process of iterative user engagement adopted in ECEM is described, from the initial presentation of the team's first vision for such a service to support external stakeholders, through to evaluation of the final interactive tool for visualisation, data download and supporting documentation (the C3S ECEM Demonstrator). The outcomes of this evaluation are outlined, together with a retrospective reflection on the engagement and development process. The extent to which co-production and co-design were achieved in practice is assessed. The paper also highlights the distance travelled from the start to end of ECEM in terms of building capacity, developing a community of practice, and raising the Technology Readiness Level. The relevance of ECEM for the European climate services market is briefly considered, including the development of downstream commercial services which build upon the public C3S services.

## Practical implications

The challenges associated with building a strong market for climate services, even for sectors such as energy with evident and increasing sensitivity and vulnerability to weather and climate variability and change, include generally low awareness and capacity and thus low uptake by potential users, as well as lack of appropriately tailored information. These challenges are being addressed by activities such as the European Copernicus Climate Change Service (C3S, https://climate.copernicus.eu/) with its

Sectoral Information System (SIS) contracts. The focus of this paper is European Climatic Energy Mixes (ECEM, https://climate.copernicus.eu/european-climate-energy-mixes), a C3S SIS which has successfully developed a proof-of-concept climate service for the energy sector.

User needs and requirements are highly dependent on the decision-making context and are therefore diverse, even across a 'single' sector. Thus, rather than attempting to deliver 'everything' for 'everyone', ECEM deliberately targeted the particularly climate/weather sensitive issues of electricity demand and renewable energy (wind, solar and hydropower) generation, supporting exploration of the demand-supply balance and its

<footnote><sup>a</sup>School of Environmental Sciences, University of East Anglia, Norwich NR4 7TJ, United Kingdom</footnote>

<footnote><sup>b</sup>World Energy & Meteorology Council, The Enterprise Centre, University Drive, University of East Anglia, Norwich NR4 7TJ, United Kingdom</footnote>

<footnote><sup>c</sup>Met Office, FitzRoy Road, Exeter EX1 3PB, United Kingdom</footnote>

<footnote><sup>d</sup>EPhysLab & CIM UVigo, Universidade de Vigo, Campus As Lagoas, 32004 Ourense, Spain</footnote>

<footnote><sup>e</sup>Department of Meteorology, University of Reading, Reading RG6 6BB, United Kingdom</footnote>

<footnote><sup>f</sup>National Centre for Atmospheric Science, University of Reading, Reading RG6 6BB, United Kingdom</footnote>

<footnote><sup>g</sup>ENEA Climate Modelling Laboratory, Bologna, Italy</footnote>

<footnote><sup>h</sup>EDF R&D, Weather, Climate and Renewable Energies Forecasts Group, Palaiseau, France</footnote>

<footnote><sup>i</sup>Institute for Environmental Analytics, Philip Lyle Building, Whiteknights Campus, Reading RG6 6BX, United Kingdom</footnote>

<footnote><sup>j</sup>MINES ParisTech, PSL University, OIE - Center Observation, Impacts, Energy, CS10207, 1 rue Claude Daunesse, Sophia Antipolis, cedex 06904, France</footnote>

<footnote><sup>k</sup>TRANSVALOR S.A., E-Golf Park, 950 avenue Roumanille, CS 40237, 06904 Sophia Antipolis cedex, France</footnote>

<footnote>* Corresponding author.
E-mail address: c.goodess@uea.ac.uk (C.M. Goodess).</footnote>
sensitivity to climate drivers. Having clearly identified the main subject focus, six categories of users and potential users were specified: (i) transmission and distribution system operators, (ii) energy generation companies, (iii) national and regional depart- ments/agencies, (iv) international associations and coordination activities, (v) policy and operational research organisations, and (vi) commercial climate service providers for the energy sector. The latter category turned out to be a more important group than anticipated at the start of ECEM, illustrating that not all users are just ‘end users’ and that commercial activities are an essential component of the market adding value to the climate services chain. Having a clearly defined target audience made it easier to manage expectations and to delimit the scope of the service. This was particularly important within the constraints of the proof-of-concept contract, which aimed to develop a pre-operational, ra- ther than a fully operational, service.

The final evaluation and feedback on the ECEM products and outcomes was generally positive, indicating that ECEM has helped to close the usability gap and delivered data and in- formation that is credible, legitimate and salient. The ECEM team attributes this success to having had a good idea of the main target and structuring all user engagement activities to address this target. From the outset, the team of six institutions (including universities, a national weather service and an energy company) had a clear vision of what they wanted to achieve and a plan of the main direction of travel. This was based on their under- standing of the potential new opportunities which climate ser- vices offer the energy industry and previous experience in pro- ducing data sets likely to be relevant for this community. The team did not, however, attempt to impose their vision on external stakeholders but instead devised an interactive and evolving process of stakeholder engagement. This meant that the team was flexible and dynamic and able to respond and adapt to users' expectations and use cases. Whether or not the team fully achieved its stated ambition of ‘co-production’ can be debated – as can the meaning of such terms in practice. Nonetheless, it is concluded that having a good process of stakeholder engagement led to both good outputs and good outcomes.

The proof-of-concept climate service developed by ECEM en-
compasses three inter-related elements. The first and most ob-
vious ‘product’ is the C3S ECEM Demonstrator (http://ecem.
wemcouncil.org/). This interface allows users to visualise, ex-
plore and download climate and energy data sets on three dif-
ferent timescales (historical, seasonal forecasting and climate
projections) aggregated to the country or, in some cases, sub-
country, level. The second element is high-quality climate and
energy data which can be used to inform an end-user problem
with or without further tailoring by, for example, a commercial
service or consultancy, or by in-house experts in an energy
company. Data consistency and quality were highlighted as es-
sential requirements of a climate service in the stakeholder eva-
luation and feedback. These data are embedded within the
Demonstrator and can be downloaded in text format which was
identified as the ‘first choice’ for much of the target audience.
However, ECEM also identified a group of more technical users
who are primarily interested in the data alone. For this group, a
facility was provided to directly download by FTP large volumes
of underlying gridded data in NetCDF format. The third element
is ‘know-how’ about how to use climate information as actionable
information. This was underpinned by a range of documentation
and guidance embedded within the Demonstrator, including Key
Messages and Event Case Studies, as well as a programme of
webinars and workshops/symposia.

All three elements (the Demonstrator, data and ‘know how’) were assessed as important by ECEM stakeholders, although different individuals or organisations naturally had different priorities in terms of what was most important for them. Similarly, some Demonstrator users were more interested in climate than energy data, or in say the historic period rather than seasonal forecasting or climate projections. Thus, the Demonstrator was designed with multiple entry points – also recognising that decision making is not a linear process. At the same time, one of the

greatest strengths of the ECEM team was its capacity to produce
consistent and high-quality data for all three timescales, for both
energy and climate, and then to integrate all into a single inter-
face to enable seamless and transparent decision making across a
business. The Demonstrator can therefore also be viewed as a
‘shop window’, which may attract a user in search of a particular
product – but who may then explore other products on offer.

A key element in ECEM has been the compilation of a robust energy database, consisting of energy demand as well as generation from wind, solar and hydro power, together with information on installed generation capacity, for all EU countries. Such a database has recently been made available by the European Network of Transmission System Operators for Electricity (ENTSO-E) under an EU regulation, but these data present some issues and inconsistencies, which made their use difficult in ECEM. The point here is not to criticize the ENTSO-E data, rather to emphasize that significant progress needs to be made on energy data completeness and homogenization in order to develop operational, fit for purpose climate services for the energy sector.

In practice, it was not so easy to treat seasonal forecasting information, which requires displaying more frequently updated data, in the same way as the historical or projection information. Issues relating to the generally limited and specific skill over Europe, probabilistic formats and user expectations and use for operational decision making also emerged. During the final symposium, for example, many users were most interested in seasonal forecasts because they had less existing awareness and understanding of these elements and saw the Demonstrator as vital to understand what they are and how to potentially use them. Thus the expectations of these users were rather different to those for whom ‘Data is King’.

Through a carefully planned process of iterative engagement with stakeholders, ECEM has raised the capacity of both providers and users to deliver and use climate services for the European renewable energy sector. The emerging community of practice is now focused on spin-off activities encompassing development of an operational service (C3S Energy – https://climate.copernicus.eu/operational-service-energy-sector) as well as commercial products and services, and research, for example on the improved tailoring and assessment of the added value of seasonal forecasts.

1. Introduction

The development of a market for climate services in Europe is the motivation for substantial investments in regional, national and private sector initiatives such as the European Union's European Roadmap for Climate Services (E.C., 2015). The context for these initiatives is the recognition of the need to better understand both the supply (i.e. provider or “push”) and demand (i.e. user or “pull”) perspectives together with an aspiration towards co-design, co-production and co-evaluation (Schuck-Zöller et al., 2017; Street, 2016; Vaughan and Dessai, 2014). A strong motivation for these investments is to mitigate threats, for example potential increases in energy costs due to the higher balancing charges due to less predictability in the system. At the European level, the Copernicus Climate Change Service (C3S – https://climate.copernicus.eu/) was launched in 2015 to lead and co-ordinate development of climate service infrastructure and underlying data provision. Amongst the first C3S activities to be funded were a number of Sectoral Information System (SIS) projects with the aim of producing proof-of-concept climate services. Two of the first SIS contracts targeted the energy sector: CLIM4ENERGY (http://clim4energy.climate.copernicus.eu) and, the focus of this paper, the European Climatic Energy Mixes (ECEM – https://climate.copernicus.eu/european-climate-energy-mixes) contract which ran from November 2015 to March 2018 (Troccoli et al., 2018).

The energy sector is sensitive and potentially vulnerable to weather
and climate, both in terms of climate variability and climate change
<table><caption>**Table 1**
The ECEM stakeholders: (a) ECEM project team members and (b) External stakeholders.</caption><thead><tr><th colspan="2">(a) ECEM Team members</th></tr><tr><th>Institution</th><th>Key technical skills and research expertise of team members</th></tr></thead><tbody><tr><td>UEA: University of East Anglia, UK</td><td>Co-ordinator; Communication and stakeholder engagement; Historical climate data</td></tr><tr><td>EDF: Electricité de France R&D, France</td><td>Energy variables, particularly electricity demand and wind energy generation</td></tr><tr><td>MO: Met Office, UK</td><td>Seasonal climate and energy forecasting</td></tr><tr><td>ARMINES: MINES ParisTech/Armines, France</td><td>Climate and Energy variables, particularly solar</td></tr><tr><td>UReading: University of Reading, UK</td><td>Event Case Studies; Software and visualisation development for the Demonstrator (Institute for Environmental Analytics)</td></tr><tr><td>ENEA: Agency for new technologies, energy and sustainable development, Italy</td><td>Energy variables, particularly hydro; Seasonal energy forecasting</td></tr></tbody><thead><tr><th colspan="2">(b) External stakeholders</th></tr><tr><th>Category</th><th>Examples of organisations actively participating in ECEM</th></tr></thead><tbody><tr><td>Transmission System Operators (TSOs) and Distribution System Operators (DSOs)</td><td>ENTSO-E, Fingrid, National Grid, RTE</td></tr><tr><td>Energy companies</td><td>EDF, EnBW, Endesa, Enel, Eon, Naturgy/Fenosa Chair, Statkraft, Vattenfall</td></tr><tr><td>National and regional departments/agencies</td><td>EC DG-Joint Research Centre, European Environment Agency</td></tr><tr><td>International associations and coordination activities</td><td>Global Framework for Climate Services</td></tr><tr><td>Policy and operational relevant research organisations</td><td>CEA/CNRS, Fraunhofer ISE, University of Leeds</td></tr><tr><td>Commercial climate and service providers for the energy sector</td><td>Climbiz, Transvalor, Vortex, Weatherquest</td></tr></tbody></table>

(Ebinger and Vergara, 2011; Johnston et al., 2012; Schaeffer et al., 2012; Troccoli, 2018). This is particularly the case for renewable energy (namely, wind, solar, hydropower) generation and electricity demand. There is growing recognition, for example, that five-years of wind power output where new farms are being installed is not sufficient to understand extreme events, and thus longer historical series are needed for both climate and generation (Cannon et al., 2015). Shifting energy policy towards decarbonisation is catalysing the growth of renewables across Europe. Hence this weather/climate sensitivity is likely to increase with the increasing contribution of renewable energy to electricity generation, and increasing electricity demand, for air conditioning for example (Troccoli et al., 2018).

ECEM was developed in response to these factors, which are affecting the need for climate services, as reflected in the emerging demand from the energy sector (Bruno Soares et al., 2018). The importance of the energy sector in terms of climate services is also reflected by its selection as one of the ‘exemplar’ sectors for the Global Framework for Climate Services (WMO, 2017). In the case of seasonal forecast information, the earlier EU-funded EUPORIAS project (Buontempo and Hewitt, 2018), concluded that although uptake of this information is generally fairly limited across Europe, the energy sector does include some so-called ‘early adopters’, including Electricité de France (EDF) (Soares and Dessai, 2015). In general, however, the wider use of such climate information by the energy sector was limited at the start of ECEM.

ECEM also started against a backdrop of growing evidence of the potential to produce skilful seasonal forecasts for the European energy industry and other sectors (Clark et al., 2017; Doblas-Reyes et al., 2013; Palin et al., 2016), as well as studies indicating the potential impacts of anthropogenic climate change on European energy demand (Damm et al., 2017; Wenz et al., 2017) and solar (Jerez et al., 2015), wind (Tobin et al., 2015) and hydropower (van Vliet et al., 2016) generation. Like its sister SIS contract CLIM4ENERGY, ECEM was innovative in aiming to bring together information on both these timescales, i.e. seasonal forecasting – a season ahead, and climate projections – out to the middle or end of the century, together with historical information for the last 40–50 years.

Rather than attempting to address the whole energy sector, ECEM deliberately focused on the particularly climate/weather sensitive issues of electricity demand and renewable generation and two related challenges which were highlighted in the original proposal:

* To describe the ways in which electricity supply and demand over Europe are affected by the spatial and temporal variations of their climate drivers

* To produce scenarios that demonstrate how different energy supply mixes can meet demand at the European scale, particularly given the projected high level of highly climate-sensitive renewable energies.

Implicit in these issues and challenges is the objective to provide consistent climate and energy data which is readily applicable to the decision-making context. These focal points also helped to constrain the target audience and potential users of the ECEM proof-of-concept climate service and thus, right from the start of the contract, determined the specific organisations and individuals with which the project team sought to engage in order to achieve its ambition of co-production and co-design. Thus organisations such as investment banks involved in the risk analysis and market pricing of electricity were not explicitly targeted, although they too have a need for climate information.

The target audience of ECEM was rather different from that of CLIM4ENERGY which focused on a number of application case studies each involving specific end users. Thus the process of stakeholder engagement was also rather different in the two projects. The CLIM4ENERGY approach provided scope for more in-depth and specific tailoring for each case study, but could be considered less extendable towards a public operational climate service (see Section 5).

The development of the new and improved climate and energy data sets powering the ECEM proof-of-concept is described elsewhere (Bett et al., 2018a, 2018b; De Felice et al., 2018; Dubus et al., 2019; Jones et al., 2017; Saint-Drenan et al., 2018; Troccoli et al., 2018). Here the focus is on the process of engagement with users and stakeholders, particularly in the context of development of the C3S ECEM Demonstrator. The Demonstrator (http://ecem.wemcouncil.org/) comprises an online interactive tool to display, explore and download climate and energy data for three timescales (historical, seasonal forecasting and climate projections) along with supporting user documentation and guidance. Throughout this paper the term ‘stakeholders’ is used to refer to both the ECEM project team and external stakeholders i.e. external users and potential users of the Demonstrator and other ECEM outputs (see Table 1).

According to the original proposal, ECEM aspired to ‘produce a relevant and useful user-driven and scientifically robust end-to-end proof-of-concept climate service for the energy sector’ and thus sought ‘a close
engagement of knowledge co-production with users to assess their priorities and requirements ... with reiteration and feedback as the project develops'. While the extent to which these ambitions were being achieved was reviewed on an ongoing basis throughout the project, this was not done within a formal assessment framework. Furthermore, the ECEM consortium did not specifically discuss what was understood by the terms 'co-production' or 'co-design'.

For the purpose of this paper, co-production is considered as a sustained collaborative process between scientists and decision makers for the production of useful, actionable and socially robust knowledge (Beier et al., 2017; Meadow et al., 2015; Taylor et al., 2017). Meadow et al (2015), for example, define co-production of knowledge as 'the process of producing usable, or actionable, science through collaboration between scientists and those who use science to make policy and management decisions'. As well as outlining the emergence of the concept of co-production over the last decade or so they highlight the potential benefits that come from framing and answering research questions together, including a greater sense of ownership over the final product.

This paper thus represents a retrospective reflection on ECEM structured around a somewhat clearer understanding of 'co-production' and some of the climate service challenges and research issues identified in the recent literature. These research issues include three priorities with respect to the Global Framework for Climate Services identified by Vaughan et al. (2016): 1) understanding users' needs, contexts and capacities; 2) improving communication; and, 3) tailoring of information. Addressing these research priorities should help to improve the credibility, legitimacy and salience of the information and products underpinning climate services (McNie, 2013; Vaughan and Dessai, 2014) and thus help to close or at least narrow the so-called usability gap (Lemos et al., 2012). In this context, it is helpful to distinguish between evaluation of the actual process of co-production and evaluation of the outputs and outcomes (Schuck-Zöller et al., 2017). Other challenges include evaluating the quality and effectiveness of stakeholder engagement (Gardiner et al., 2018; Hewitt et al., 2017; Schuck-Zöller et al., 2017; Swart et al., 2017) and developing and sustaining communities of practice (Street, 2016; Vincent et al., 2018). In relation to development of the climate services market, the European Roadmap for Climate Services identifies three challenges: enabling market growth; building the market framework; and enhancing the quality and relevance of climate services (E.C., 2015).

Using these issues and challenges as a structural framework, the self-assessment and reflective process described in this paper addresses the following questions:

* How successful was ECEM in its aim of co-design/co-production?

* Did the process of stakeholder engagement lead to higher quality outputs/products and outcomes than might be expected in the absence of such engagement?

* To what extent has ECEM contributed to capacity building and market development within the European energy sector climate services community?

The paper is structured as follows. Section 2 on Methods describes how the consortium was built and developed to fit the scope and focus of ECEM and the timeline of iterative engagement and Demonstrator development as well as the engagement, feedback and evaluation mechanisms used. Section 3 outlines the main outputs of ECEM focusing on the Demonstrator and identification of what is distinctive about it and the embedded data, as well as its evolution given the design process. Section 4 discusses the engagement processes and outcomes focusing on the three questions above. The concluding Section 5 highlights the distance travelled from the start to the end of ECEM and outlines how the work and expertise is being taken forward.

## 2. Methods

The ECEM project team (listed in Table 1a) included research organisations and individuals who already had some expertise in processing and assessing climate and energy variables on the three target timescales of historical, seasonal forecasting and projections. The majority of team members already had some experience of working with the renewable energy sector and varying levels of understanding of the types of data and climate information required by the target audience for decision making (see Section 1). The team also included an organisation, the Institute for Environmental Analytics at the University of Reading, with the expertise in visualisation and online portal development required for the Demonstrator. The team members most closely involved in designing and implementing the stakeholder engagement activities were able to draw on the experience of, amongst others, the earlier EUPORIAS project (Soares and Dessai, 2015, 2016) and involvement in climate service network activities such as the Climate Services Partnership and its series of International Conferences on Climate Services. The team included organisations involved in both public and commercial operational weather services (e.g. the UK Meteorological Office, MO) and three founding Directors of the World Energy and Meteorology Council including its Managing Director (who was also the ECEM project co-ordinator).

While the diversity of users and the need to tailor climate services for specific sectors is widely recognised (Bruno Soares et al., 2018), and motivated the C3S SIS initiative, it is also the case that decision making needs are diverse within a particular sector. The nature of the relevant decision making also differs depending on the timescale of the information provided (see, for example Figure 7 of Bruno Soares et al. (2018)). Seasonal forecasting timescales are relevant for operational, trading and maintenance activities, for example, while climate projection timescales are relevant for the wider vision and strategy of an organisation, such as making decisions around infrastructure investment. The focus of ECEM on these two timescales, along with historical (the last few decades), was dictated by C3S. Thus weather forecasting and subseasonal forecasting, although relevant for operations, were explicitly excluded due to the C3S remit. Initialised decadal predictions were not considered as these are not yet available, for example through C3S. The omission of some timescales may, however, appear rather arbitrary for those organisations who employ a number of different timescales in different sections of the business (Bruno Soares et al., 2018).

The scope and focus of ECEM was thus constrained by the wider context of the C3S SIS programme as well as by the resources and time available for development of a proof-of-concept climate service. In view of these constraints, a decision was taken at the proposal stage to focus on the country scale rather than attempting to go straight to higher spatial resolution. Flexible methodologies for data and Demonstrator development were however devised which would make it relatively easy to move to higher spatial (and temporal) resolution in the future, should the necessary energy data be available. Some sub-country information, for example, was eventually produced by ECEM – see Section 3. The lack of a European-wide, homogeneous and sufficiently long energy dataset was indeed a major challenge in ECEM. These issues relating to scope and focus were important in terms of managing user expectations throughout the project and in the identification of appropriate target users.

Considering all the above issues, one of the early tasks of ECEM was to use the strong existing networks and expertise of team members accrued over many years to identify the target categories of external stakeholders (i.e., potential users) and relevant organisations in each of these categories. Six such categories were identified as listed in Table 1b along with examples of organisations in each category who actively participated over the course of ECEM (e.g., as workshop and webinar attendees, and Advisory Board members). The six categories include energy generation companies such as EDF. EDF R&D was also
<figure><img src="image_5.png" /><figcaption>**Fig. 1.** Schematic of ECEM stakeholders and actors and their inter-relationships. The black arrows in the central boxes and on the right-hand side of the figure show the evolution of these relationships over time. C3S: Copernicus Climate Services. SIS: Sectoral Information System. For ECEM partner abbreviations see Table 1a. TSO/DSO: Transmission/Distribution System Operators. WEMC: World Energy and Meteorology Council. The grey arrows on the left-hand side of the figure show the increase in Technology Readiness Level (TRL – Heder, 2017) over time. TRL 2: technology concept formulated, TRL3: experimental proof of concept, TRL4: technology validated in lab, TRL5: technology validated in a relevant industrial environment, TRL6: technology demonstrated in a relevant industrial environment, TRL7: system prototype demonstration in an operational environment, TRL8: system complete and qualified, TRL9: actual system proven in operational environment.</figcaption></figure>

part of the project team demonstrating that the same organisation can act as both a provider and user of climate services (Table 1). As noted in the Introduction, EDF are considered as one of the ‘early adopters’ of climate information in the energy sector and the EDF R&D team plays an important role in providing climate and climate-based energy information (services) to other parts of this large organisation. The EDF R &D team itself can also be considered as a user of climate data provided by other members of the ECEM team.

Fig. 1 illustrates the relationship between the project partners and the external stakeholders (Table 1) and the development of these stakeholder relationships over the course of the 29-month project. The ECEM level of achievement is discussed in Sections 3 and 4, while the advances in terms of Technology Readiness Levels (TRL – left-hand arrows in Fig. 1) (Heder, 2017) are discussed in Section 4 together with the progression from a disconnected community of potential users towards a stronger and more coherent community of users and target users (right-hand arrows). Here, the importance of new users and champions who became more actively involved with the project team over the course of the project is noted. A specific example of how active participation in ECEM is leading to further downstream development and commercial application of some of the ECEM outputs is presented in Box 1.

Fig. 2 shows the intertwined timeline of Demonstrator development and stakeholder engagement highlighting the three ECEM stakeholder workshops and two C3S Symposia on Climate Services for the Energy Sector that were jointly organised by ECEM and CLIM4ENERGY. These workshops and symposia were the key engagement events used by ECEM but were complemented by a range of other mechanisms designed particularly to maintain contact and momentum in between these face-to-face events (Table 2).

Prior to the first ECEM stakeholder workshop in February 2016, only general specifications or ‘first vision’ for the Demonstrator were available and presented to the stakeholders. It was a deliberate decision, in order to avoid constraining ideas, not to develop a visual tool prior to first interactions with potential users. As well as discussing the energy and climate variables required and the potential functionality of a Demonstrator, the workshop particularly considered decision-making aspects through the development of ‘user stories’. For this exercise, participants were arranged into breakout groups according to their user group or category (see Table 1b). Each group was then asked to construct ‘user stories’ in the form “AS a < user type > I WANT TO < do something > OR TO < have access to x information/data > SO THAT I CAN < achieve something >”. The interactive and participatory nature of this workshop was also enhanced by employing a graphic cartoonist to provide a ‘live’ visual summary of discussions (available from: http://www.wemcouncil.org/Projects/ECEM/Copernicus_ECEM_logos_80cm.jpg). This provided a lively magnet and an evolving talking point helping participants to get to know each other better. Elements of the cartoon were subsequently used throughout the project for communication purposes, particularly in presentations and promotional material.

Having received general endorsement from the first workshop, work then focused on developing the Demonstrator logic, i.e., the requirements for the landing page, display functionalities and menu choices and variants. This logic document was then used to develop the first visual version of the Demonstrator wireframe which was shown to participants during the second stakeholder workshop in June 2016. This first version of the Demonstrator focused on the display of historical and projected temperature only but nonetheless allowed potential users to see and evaluate how the Demonstrator might look and
<figure><img src="image_6.png" /><figcaption>**Fig. 2.** Timeline of Demonstrator development and stakeholder engagement. SH WS: Stakeholder Workshop.</figcaption></figure>

how they could interact with the system. The carousel (i.e. rotating break-out group) sessions during the second workshop focused on the Demonstrator wireframe and climate and energy data requirements but were complemented by a participatory exercise designed to better understand the different types of users and their potential decision-making applications, as well as to inform preparatory work on the ECEM event case studies. This exercise addressed the following question: “What aspects of climate do you see as posing the greatest risk to maintaining secure supply-demand balance in the European power sector, both now and into the coming decades?”

By the time of the first Symposium in February 2017, corresponding to the third stakeholder workshop, the Demonstrator Wireframe was at an appropriate stage of development for hands-on evaluation and testing by stakeholders. The Symposium therefore included a two-hour hands-on session which started with a brief walk through the Demonstrator menu structure and functionalities. Participants were then free to explore the Demonstrator for themselves with the ECEM team on hand to answer questions about the Demonstrator itself and the data currently incorporated. The temporary link to the Demonstrator provided to participants remained live for three weeks after the symposium to support and encourage further testing and feedback.

The Demonstrator continued to be developed in an iterative approach with frequent input from stakeholders and prospective users. As well as the stakeholder workshops, symposia and webinars, one feedback mechanism that became particularly useful during the final stages of development was a feedback form link on the Demonstrator itself (Table 2). By the time of the fourth stakeholder workshop in June 2017 the Demonstrator was sufficiently advanced to support both ‘Exploring the Demonstrator’ and ‘What can the Demonstrator do for you?’ sessions. In the latter, external stakeholders worked with ECEM project team members to develop their own tailored data assessments or mini case studies, many of them focusing on specific extreme events identified by the external stakeholders and how their frequency might change in the future.

At the end of October 2017, a close to final version of the Demonstrator (the Proof-of-Concept) was made available for the subsequent five-month period of pre-operational evaluation and refinement. The engagement and feedback mechanisms listed in Table 2 continued to be used during this period – including a second and final Symposium in March 2018. In addition, the evaluation mechanisms listed at the end of Table 2 were designed and implemented during this period.

### 3. Results: outputs and products focusing on the C3S ECEM demonstrator

Table 3 summarises the climate and energy data incorporated in the ECEM Demonstrator, along with its functionality and embedded documentation and guidance. The climate and energy variables include those proposed in the first vision of the Demonstrator (see Section 2 and Fig. 2) but the need for these was repeatedly endorsed by the external stakeholders and by the EU-funded ConnectingGEO project (http://www.connectinggeo.net/). Sea level pressure was added primarily at the request of ECEM team members in order to facilitate exploration of modes of climate variability such as the North Atlantic Oscillation which in turn influence both spatial (e.g., from northern to southern Europe) and temporal (e.g., year-to-year) variability in the energy variables (Brayshaw et al., 2011; Clark et al., 2017; Thornton et al., 2017). Snow depth was added at the request of stakeholders including team members who required it for the calculation of hydropower generation. Due to concerns about data quality, however, this variable was only provided for historical timescales and not for projections (and was not available in the case of seasonal forecasting).

Although it was originally proposed to restrict the spatial scale to the country level (see Section 2) and the team sought to manage users' expectations in this respect, the desirability of having higher spatial resolution information was acknowledged and in most cases it was also possible to provide information at the level of “clusters”. These are 96 regions identified by the eHighway 2050 project based on NUTS3 regions (see https://www.entsoe.eu/outlooks/ehighways-2050/).

In terms of functionality, the Demonstrator was designed to provide an appropriate balance between online visualisation and exploration, and data download, recognising that getting this balance right is an important and complex issue for climate service development. During the first ECEM stakeholder workshop (Fig. 2) participants generally gave higher priority to data access than to visualisation. This was summarised in a key workshop message as ‘Data is King’. It was therefore decided to incorporate an ‘advanced’ data download facility within the Demonstrator. However, this prioritisation tended to reflect the generally greater technical expertise of this workshop’s participants who included a number of consultants and intermediary organisations. Indeed, the preferred format of the majority of end users is for data to be downloaded directly from maps and time-series displayed in the Demonstrator in text format (CSV files), but a link is also provided to an FTP site where ‘advanced’ users can download data at 0.5 degree
<table><caption>**Table 2**
ECEM engagement, feedback and evaluation mechanisms.</caption><thead><tr><th>Engagement mechanisms</th><th>Audience</th><th>Description</th></tr></thead><tbody><tr><td>Workshops</td><td>Open invitation – three ECEM-only events (typically 30–40 participants) and two joint symposia (see below).</td><td>Participatory elements tailored to the development stage of the Demonstrator including carousels, participatory exercises (e.g. User Stories), hands-on sessions (e.g., Build your own tailored data assessment).</td></tr><tr><td>Two C3S Symposia on Climate Services for the Energy Sector</td><td>Jointly organised by ECEM and CLIM4ENERGY: 22–23 February 2017, Barcelona (~80 participants)<br><a href="https://climate.copernicus.eu/copernicus-symposium-climate-services-energy-sector">https://climate.copernicus.eu/copernicus-symposium-climate-services-energy-sector</a><br>5–6 March 2018, Paris (~100 participants)<br><a href="https://climate.copernicus.eu/2nd-copernicus-climate-change-symposium-climate-services-energy-sector">https://climate.copernicus.eu/2nd-copernicus-climate-change-symposium-climate-services-energy-sector</a></td><td>Presentations by team members and external stakeholders, and interactive sessions on the Demonstrator.</td></tr><tr><td>Webinars</td><td>Open invitation, some non-ECEM speakers. Seven ~1 h events (typically 40–60 participants).<br><a href="http://www.wemcouncil.org/wp/european-climatic-energy-mixes">http://www.wemcouncil.org/wp/european-climatic-energy-mixes</a><br>Later made publicly available on YouTube.</td><td>Range of topics from technical presentations on underlying datasets to general introduction to the Demonstrator.</td></tr><tr><td>Advisory Committee</td><td>Nine representatives from Transmission System Operators, Energy companies, Regional agencies, Research organisations, Service providers.</td><td>Telecons and email interactions.</td></tr><tr><td>ECEM-athon</td><td>Project team, plus selected stakeholders: ~20 participants. 6–10 November 2017, Opio, France.</td><td>Week-long intensive event to assess datasets, the Demonstrator and to develop journal papers. Self-reflection on 'things we did well' and 'things we could have done better'.</td></tr><tr><td>Bi-weekly project meetings<br>Project website</td><td>Project team.<br>Open access – with links to C3S and CLIM4ENERGY.<br><a href="https://climate.copernicus.eu/european-climate-energy-mixes">https://climate.copernicus.eu/european-climate-energy-mixes</a></td><td>Regular teleconferences.<br>Provides access to webinar recordings, workshop presentations and the Demonstrator.</td></tr><tr><td>Twitter<br>Videos<br>Cartoon</td><td>Participants in workshops/symposia encouraged to tweet.<br>Three-minute promotional video on the Demonstrator.<br>A graphic cartoonist was employed to provide a 'live' summary of discussions during the first stakeholder workshop.</td><td>#C3S_ENERGY<br>Full cartoon: <a href="http://www.wemcouncil.org/Projects/ECEM/Copernicus_ECEM_logos_80cm.jpg">http://www.wemcouncil.org/Projects/ECEM/Copernicus_ECEM_logos_80cm.jpg</a>. Extracts used in various presentations and promotional material.</td></tr><tr><th colspan="3">Feedback mechanisms</th></tr><tr><td>Workshop reporting</td><td>See above.</td><td>Information from Rapporteurs, Post-it notes, Feedback forms incorporated into reports.</td></tr><tr><td>Demonstrator feedback form</td><td>Link to google form on Demonstrator.</td><td>Four questions plus free text.</td></tr><tr><td>Webinar questions</td><td>All webinar participants.</td><td>Selected questions written-up as Demonstrator FAQs.</td></tr><tr><td>Telephone surveys and one-to-ones with selected users</td><td>Semi-structured interviews with members of the Advisory Board and other key users particularly potential 'champions'.</td><td>A number of pre-defined fairly high-level questions with scope for broader discussion. Undertaken by the project manager and leader.</td></tr><tr><td>Recorded interviews</td><td>Two interviews with participants in the final Symposium – undertaken by the Project Manager.</td><td>Asked about interest in the Symposium and how they would use the demonstrator.</td></tr><tr><td>Google Sheet</td><td>Maintained by the Demonstrator developer/coordinator.</td><td>Spreadsheet used to report issues/problems with the Demonstrator including those from feedback forms and additional user needs & requests. Proposed responses entered and entries moved to 'Done' or 'Archive' sheets once actioned.</td></tr><tr><th colspan="3">Evaluation mechanisms</th></tr><tr><td>Google Analytics</td><td>Number of sessions, users, page views, session duration for the Demonstrator.</td><td>Reported to ECMWF quarterly as Key Performance Indicators.</td></tr><tr><td>Advisory Board 'final' feedback</td><td>Email and verbal.</td><td>Three questions on Demonstrator functionality/use, and also 'What has ECEM done for you?'.<br>First five questions (see Section 4) answered on a five-point scale (strongly agree to strongly disagree) and 'What do you particularly like about the ECEM Demonstrator and what could be improved?' ('free text response).</td></tr><tr><td>Short user survey</td><td>Sent to all attendees in ECEM webinars and workshops/symposia.</td><td>Two sets of questions on the participants (Day 1) and ECEM products (Day 2).</td></tr><tr><td>Interactive polls</td><td>All participants in the final symposium answered questions with a number of fixed responses in real time using <a href="http://www.sli.do">www.sli.do</a>.</td><td></td></tr></tbody></table>

latitude/longitude resolution in NetCDF format. These gridded data were aggregated to provide the country/cluster values provided in the Demonstrator itself.

In subsequent workshops and engagement activities, stronger requirements for flexible and easy-to-use visualisation capabilities were expressed, reflecting the broader range of potential users involved in these later events. Thus in the final stages of Demonstrator development (Fig. 2) more effort was channelled into the visualisation functionality including, for example, improved user control of map colour schemes. It was also decided to further support users by providing some pre-prepared graphs for seasonal forecasting skill scores and climate projections along with short guidance documents – see below. The prepared

graphs were constructed to deliver faster plotting as the plots include input from multiple data files and to illustrate the importance of adequately considering projection uncertainty. Hence these projection plots automatically include the multi-model mean, an ensemble range and individual model outputs for two different emission scenarios (Representative Concentration Pathways, RCP, 4.5 and 8.5 – (van Vuuren et al., 2011)) as well as observed data (Fig. 3). At the same time, users retain the flexibility to construct their own projection plots.

Particular consideration, especially in the later stages of Demonstrator development, was given to producing a comprehensive suite of documentation and guidance targeted at a range of users from general users and those new to climate and/or energy data through to
<table><caption>**Table 3**
Summary of the Demonstrator data, functionality and documentation/guidance.</caption><thead><tr><th>Feature</th><th>Description</th></tr></thead><tbody><tr><td>Climate variables</td><td>Temperature, Precipitation, Wind speed (10 m and 100 m), Solar radiation (GHI), Relative humidity, Sea level pressure, (Snow depth).</td></tr><tr><td>Energy variables</td><td>Demand [energy (MWh) and power (MW)], Hydro (reservoir and run of river), Solar PV and onshore Wind generation (all provided as capacity factor, energy and power).</td></tr><tr><td>Time period</td><td>Historical (1979–2016), Seasonal Forecasting, Projections (1980–2100).</td></tr><tr><td>Temporal resolution(s)</td><td>Daily, Monthly, Seasonal, Annual (Historical/Projections). Seasonal (Winter/DJF and Summer/JJA) averages for Seasonal Forecasting.</td></tr><tr><td>Spatial resolution</td><td>Countries and (except for demand and hydro) clusters (96 regions from the eHighway 2050 project, based on NUTS3 regions).</td></tr><tr><td>Historical climate data</td><td>Bias-adjusted ERA-interim.</td></tr><tr><td>Seasonal forecasting</td><td>Historical skill measures (correlation, ROC and Brier skill scores) for ECMWF, Météo-France and MO systems, winter/ summer hindcasts.</td></tr><tr><td>Climate projections</td><td>Bias-adjusted outputs from simulations by seven Regional Climate Models provided by CLIM4ENERGY for RCP4.5/ RCP8.5.</td></tr><tr><td>Representation of ensemble uncertainty</td><td>Ensemble mean and smoothed max/min range plus individual models. Pre-prepared graphs (full plots) also compare Representative Concentration Pathways – RCP4.5/RCP8.5.</td></tr><tr><td>Ability to only map ‘significant’ values</td><td>Seasonal Forecasting skill maps: option to grey-out countries with non-statistically significant skill.</td></tr><tr><td>Energy models</td><td>In general, only the model variant found to be most skilful/robust for the historical period is used.</td></tr><tr><td>Climate and socio-economic forcing (including energy mix scenarios)</td><td>Capacity factor: highlights the influence of climate variability and change. Demand: provided with the long-term economically-driven trend removed, as well as normalised anomalies and with the trend retained. Energy and power projections: for five eHighway2050 energy mix scenarios.</td></tr><tr><td>Functionality</td><td>Three menus: Main control with radio buttons/drop-boxes for making selections; Legend (controls map appearance); Help (documentation and guidance).<br>Functionality includes: Switch between map and time series; Multiple plotting windows; Download (Map, Time-series graph, Data, Advanced data); Modify map/graph appearance; Link for user feedback; Version control; Known issues; Cookies.</td></tr><tr><th>Documentation/guidance</th><th>Description</th></tr><tr><td>Using the Demonstrator</td><td>Getting started, how to navigate, outline of functionality etc.</td></tr><tr><td>Methods and Assumptions</td><td>Overviews of products and introduction to different types of data for the non-expert. Cross-cutting issues include guidance on the use and interpretation of the data and some of the underlying assumptions.</td></tr><tr><td>Key Messages</td><td>A series of six Key Messages for the European energy sector based on the analysis of data in the ECEM Demonstrator. Non-technical – two pages.</td></tr><tr><td>Pre-prepared graphs</td><td>Examples and guidance on the more complex graphs provided, including how to interpret uncertainty ranges for climate and energy projections and the seasonal forecasting historical skill scores. Non-technical – two pages.</td></tr><tr><td>Variable Fact Sheets</td><td>A series of factsheets which provide metadata for the climate (seven) and energy (four) variables. Technical documents – five to eight pages.</td></tr><tr><td>Event Case Studies</td><td>Three case studies based on actual extreme events which illustrate how the Demonstrator can be used by the energy sector to enhance understanding and support decision making. Non-technical – four pages.</td></tr><tr><td>FAQs</td><td>Covering ECEM and the data sets – based on questions raised at workshops and webinars.</td></tr><tr><td>Glossary</td><td>Climate and Energy terms.</td></tr><tr><td>About</td><td>Underlying software, version number, changes and updates, known issues.</td></tr><tr><td>Tutorials</td><td>Text-based map and graph tutorials available at start-up and from Help menu. Link to webinar videos and slides.</td></tr></tbody></table>

more technical users (Table 3). The Variable Fact Sheets, for example, provide detailed metadata including information about data processing as well as links to the original data sources for technical users. The six two-page Key Messages are intended to provide critical information about historical and projected changes in energy and climate variables as well as examples of the types of analyses and interpretations that can be drawn from the ECEM Demonstrator data and plots. They follow a standard template with three or four ‘Key message’ bullet points on the first page, a final question on ‘What does this mean for the energy sector?’ and a single core question where the data are presented and analysed. The six Key Messages are:

* How do we know Europe is warming?

* How do we know the future will be warmer?

* How do we know energy demand will change?

* Is Europe likely to become drier or wetter?

* Can changes in radiation be identified?

* What datasets are available?

The three user-motivated Event Case Studies illustrate the type of more detailed analyses that can be undertaken using the ECEM Demonstrator and the underpinning data, focusing on extreme weather events that have had a major impact on the European energy sector. The first Event Case Study focuses on conditions during winter 2009/10

which saw high power demand due to extremely cold temperatures across much of northern Europe, while the second considers how the frequency of such events might change over the coming decades due to climate change. The third Event Case Study considers the impact of summer heat waves on electricity supply and demand in Spain. Each of the Event Case Studies follows the same structure. The first page provides a summary in terms of: Boosting Decision Making; Scientific/Technical Advances; and, Key Lessons. The subsequent pages take a narrative approach from the initial starting point or event through the analysis and lessons learned, also illustrating ‘How can we use the demonstrator to learn more?’.

The stakeholder engagement highlighted the different types of energy sector user and the different types of decision that a user may be focusing on at any particular point in time. Thus the Demonstrator supports a number of different entry points to the data and information provided (Fig. 4). A user can choose to start exploring energy data rather than climate data, or the seasonal forecasting rather than historical or projection timescale. They can focus on the map view, or on time-series plots for individual countries/clusters. A policy maker might go straight to the Key Messages or Event Case Studies in the help and information section, whereas a consultant might go straight to the ftp data link. These multiple entry points provide flexibility and recognise that decision making is not a linear process, hence a more modular approach is appropriate (Haße and Kind, 2018; Laudien et al., 2018).
Energy Projection | Greece | Summer (JJA/Jun-Aug)

<figure><img src="image_7.png" /><figcaption>Fig. 3. Example of a pre-prepared projections graph from the ECEM demonstrator. Summer energy demand in Greece shown as power (MW) with socio-economic trends removed (see http://ecem.wemcouncil.org/pdf/ECEM Prepared Graphs Energy 20180220.pdf for details).</figcaption></figure>

On the other hand, the number of choices available to users may be daunting. Thus good guidance, such as the ‘Getting started’ part of the Demonstrator help menu, is essential.

The ability to download data in text or NetCDF format extends the power of the Demonstrator beyond a ‘simple’ visualisation tool. It gives the user flexibility to integrate the ECEM data sets with their own datasets and statistical/modelling tools. This is something else which emerged strongly from the ECEM workshops, webinars and Advisory Board discussions. Generation data, for example, is provided as capacity

factors as well as in terms of power or energy, also depending on a prescribed future energy mix taken from the five eHighway2050 scenarios (http://www.e-highway2050.eu/). A user can, however, download the capacity factors and apply them to their own or any alternative energy mix scenarios.

From the early stages of engagement with potential users of the ECEM Demonstrator it was evident that access to high-quality climate and energy data was a priority. Thus it was important to assess the quality of the ECEM outputs, focusing on the Demonstrator and the

<figure><img src="image_8.png" /><figcaption>Fig. 4. Screenshot from the ECEM Demonstrator showing the user control menu on the left-hand side, the help and information menu link at the bottom of the left-hand side, the FTP-download link on the right-hand side and an example simple time-series plot of historical winter wind capacity factor for France overlaying the country-level map.</figcaption></figure>
credibility, legitimacy and salience of its embedded data. These issues are all important to build trust in the data so that decisions can be made in the business context. With respect to salience, the majority of user requirements in terms of climate and energy variables were largely met (see Table 3), with only a few exceptions (see Section 4). The main aspect that users would like to see improved relates to higher temporal and spatial resolution.

Consistency was also identified as a key user requirement in terms of data credibility and salience and was achieved, so far as possible, with respect to a number of aspects of the data. In particular, they are consistent in terms of spatial and temporal scale and coverage, format and underlying assumptions between countries/clusters, variables, energy and climate, time period and climate/energy scenarios.

The quality of the data in terms of credibility and legitimacy was maximised by using the most reliable input data available and through processing such as bias adjustment of the climate data used to calculate energy variables. This is documented in underlying journal papers (De Felice et al., 2018; Dubus et al., 2019; Jones et al., 2017; Saint-Drenan et al., 2018; Troccoli et al., 2018) and summarised in the Variable Fact Sheets. As well as evaluation and checking by the individual data providers, all ECEM team members were able to review and assess datasets in a ‘test’ version of the Demonstrator before being made public. A number of issues were thus identified, resolved and updated data sets provided. Where specific issues could not be resolved they were documented in the ‘Known issues’ section of the Demonstrator help menu.

A question on data quality was asked as part of the final short user survey (see Table 2). This question was deliberately left somewhat ambiguous and it was evident that some respondents interpreted quality as referring to the spatial and temporal resolution and the types of variables provided (salience) – whereas others highlighted the reliability and provenance of the data (credibility and legitimacy). A few respondents wanted data at the city or catchment scale, for example – see the end of Section 4 for a brief discussion on ‘unmet’ needs. Nonetheless it is encouraging that two-thirds of the 75 respondents ‘agreed’ or ‘strongly agreed’ that the quality was appropriate for their needs. Rather than this ad hoc and post hoc approach to assessing data quality, ECEM could, for example, have developed a specific data quality matrix of energy industry requirements – which might be different from those of the scientific community.

## 4. Discussion on processes and outcomes

One of the stated objectives of the assessment reported here (see Section 1) is to consider how successful ECEM was in its aim of co-production/co-design. Such aspirations have become increasingly common over the last few years in the context of applied climate research and climate services (Beier et al., 2017; Taylor et al., 2017; Willyard et al., 2018). In general, however, terms such as co-production are used without a specific definition or in-depth discussion of their practical meaning (Meadow et al., 2015) or how to evaluate success (Durose et al., 2018). This was the case for ECEM, which nonetheless devised an evolving and iterative process of stakeholder engagement from the early stages of Demonstrator development (Fig. 2). Concepts such as scientific and social robustness, which are considered to underlie co-production of knowledge (Taylor et al., 2017), were not explicitly used within ECEM.

Similarly, although ECEM strove for meaningful engagement with users and potential users, the level of engagement sought was never explicitly defined. If the three broad categories of engagement proposed by Hewitt et al. (2017) are considered, then ECEM certainly went well beyond passive engagement and information provision (category one). Interactive group and dialogue-based activities (category two) were widely used in the workshops (Table 2) and, to a large degree, the engagement was active and led to the development of focused and targeted relationships (category three) (Hewitt et al., 2017). The

engagement was also sustained and synergistic, which is considered one of the defining characteristics of co-production (Laudien et al., 2018; Meadow et al., 2015). It is therefore concluded that ECEM moved a long way towards co-production within a relatively short period even though it takes time for people to appreciate and uptake climate information, and to grow a strong stakeholder base.

Another objective of this post-hoc assessment is to evaluate the ‘quality of relationship’ between the ECEM developers and users (Gardiner et al., 2018), and to consider whether the process of stakeholder engagement led to higher quality outputs and outcomes from ECEM than if a purely consultative approach (Meadow et al., 2015) had been followed. The issue of data quality (i.e. quality of outputs/products) is discussed in Section 3. Here, the issue of quality of outcomes (Schuck-Zöller et al., 2017) is considered, including evaluation of the Demonstrator functionality, usability and usefulness. The 76 respondents to the short user survey (Table 2) undertaken during the pre-operational evaluation period (Fig. 2) were generally positive in this respect, i.e. they ‘agreed’ or ‘strongly agreed’ with the survey statements. Over 80% of respondents were positive that the Demonstrator is ‘easy to use’ and 73% were positive that the documentation and guidance provided ‘will help you to make best use of the C3S ECEM Demonstrator’. It was particularly encouraging that 70% of respondents had already ‘identified some specific features of the C3S ECEM Demonstrator and datasets which will facilitate my work’. Feedback from this survey and the other feedback and evaluation mechanisms used in ECEM (Table 2) support the conclusion that ECEM has successfully narrowed the usability gap (Lemos et al., 2012) by developing a proof-of-concept Demonstrator that is user friendly in terms of both tool functionality and supporting documentation. It is unlikely that such positive feedback would have been obtained without iterative user engagement from the start.

The third objective here is to assess the extent to which ECEM has contributed to capacity building and market development within the European energy sector climate services community. An indication of the movement in these respects is given in Fig. 1. The process of collaboration between the ECEM partners and external stakeholders, and the relationship between this sustained engagement and co-production, are discussed above. Through this process, new users and champions, have emerged. It is estimated that the Demonstrator attracted an average of around 60 new users per month during the later stages of ECEM. Box 1 provides one example, that of Transvalor, to illustrate how a commercial product can be developed from a public climate service.

### Box 1: Working with commercial providers – the example of Transvalor

Transvalor is a commercial energy service provider (Table 1b), based in the south of France. Transvalor and ARMINES (an ECEM partner) had had some previous contact and interaction due to a shared interest in solar radiation and their close geographical location. A representative from Transvalor attended the fourth stakeholder workshop and the ECEM-athon in 2017 (Table 2). For the ECEM team, it was very informative to have time for detailed discussion with a new user, particularly with respect to climate/energy projections, including terminology and the complexity of information and uncertainty. It was agreed that it was more appropriate for Transvalor's customers to access ECEM outputs including the Demonstrator and data sets via Transvalor, after appropriate processing and repackaging, rather than directly from ECEM. A strategy by which Transvalor could gradually introduce customers to the concepts of climate projections was devised, based around the existing SoDa website (http://www.soda-pro.com/). Transvalor are now receiving more inquiries from customers concerning future conditions. Transvalor also reviewed and substantially improved the Key Message on solar radiation changes (see Section 3). During the ECEM-athon discussions, Transvalor identified a requirement for additional processing of
the ECEM radiation data sets for input to in-house modelling tools, and this work was subsequently undertaken by a MINES ParisTech PhD student. This example (others include that of Naturgy and wind/hydro generation) illustrates how commercial products and services can be built upon a 'public good' free service such as C3S (see Section 5).

The goal of ECEM – consistent with the objectives of the C3S SIS (see Section 1) – was to develop a pre-operational tool (the Demonstrator) and to implement a proof-of-concept service, i.e., one bringing together climate and energy data in a useful and usable way for the energy sector (Troccoli et al., 2018). This goal was successfully achieved (Fig. 1). Inevitably the C3S ECEM Demonstrator lacks some of the characteristics of an operational service. In particular, real-time seasonal forecasts are not yet embodied – only information about historical performance of the forecasting systems. Although the team continues to be responsive to feedback (Table 2) there is not an active ‘help desk’, typical of an operational system, responding to inquiries within service level agreement time frames. Nonetheless ECEM has moved a considerable way towards an operational system which is reflected in a retrospective assessment of Technology Readiness Levels (TRL) as adopted by the EU for the Horizon 2020 work programme (Heder, 2017). The arrows down the left-hand side of Fig. 1 indicate the estimated TRL at the start (TRL 2–4) and end (TRL 5/6) of ECEM. The starting point of individual team members varied somewhat but in general all had already studied and tested the proposed methods and main data sources in different contexts (i.e. TRL (2) technology concept formulated, (3) experimental proof of concept, and (4) technology validated in lab). By the end, the technology had been validated (TRL 5) and demonstrated (TRL 6) in a relevant industrial environment.

In order to progress from TRL 2–4 to TRL 5–6 a number of technical challenges had to be met. These challenges included bias adjustment and validation of historical ERA-interim solar radiation and 10 m/ 100 m wind data (Jones et al., 2017) and calibration/validation of generation models particularly for wind power (Dubus et al., 2019), solar power (Saint-Drenan et al., 2018) and hydro power (De Felice et al., 2018). Sourcing appropriate data for the energy models was also challenging (Dubus et al., 2019). Availability of energy data was an issue for calculating generation and demand, together with issues about how to incorporate non-climatic influences such as day of the week and technological/social trends (Dubus et al., 2019).

As a proof-of-concept climate service, one of the challenges of ECEM was to link meteorology/climate expertise to energy expertise. At times, this introduced some tension between the kinds of data that are most useful to (a) the end user and (b) the researcher. The choice was made, for example, to account for evolving trends and detailed factors such as weekends and public holidays when modelling demand data. This makes the data more directly relevant and familiar to end users, but harder to work with from a research perspective even when the research itself seeks to inform end users.

Whereas the meteorological community has adopted a data sharing and common format approach since its infancy, the energy sector situation regarding data is very different and less coordinated. Little regulation and few standards exist, which make it extremely difficult to gather a harmonized dataset at the European level. EU regulation 543/2013 imposed a new paradigm, but is very recent, and much work is still needed to reach the necessary level, even though considerable progress has already been made. Good quality and accessible data are essential to both build and validate the energy models. The major challenges in energy data are:

* The lack of standards (data formats, accuracy, quality, availability of metadata)

* The relatively short duration of available time series (for instance, hourly energy generation from wind, solar and hydropower for all

EU countries are only available since 2015)

* Inconsistencies in datasets (for instance, the net generation capacity reported on the ENTSO-E Transparency Portal (https://transparency.entsoe.eu/) is not coherent with the generation data for some countries for solar and hydropower)

The compilation and preparation of the energy data used in ECEM required a very significant amount of time and effort. In order to improve this important aspect of energy climate services development, it is essential that the above challenges are addressed in a long-term approach. This certainly requires a concerted effort, involving EU and national regulation authorities, network operators, energy companies and climate services developers. On top of ENTSO-E's Transparency Portal after EU Regulation 543/2013, several initiatives have developed to progress on data issues. These include for instance the Open Power System Data Platform (https://open-power-system-data.org) and the World Energy and Meteorology Council Special Interest Group on Data Exchange, Access and Standards; but a higher level commitment is needed, as research groups do not have the legitimacy to impose rules on the whole sector.

Addressing these challenges has substantially increased the technical capacity of the scientific team. At the same time, ECEM has helped to build the capacity of external stakeholders through ongoing exposure to relevant terminology and data, and participation in ECEM workshops and webinars. The range of Demonstrator documentation and guidance (Table 3) was also designed to build the capacity of users, whether new to climate data or more experienced.

As well as building technical capacity, the ECEM activities have led to improved individual and collective understanding of the needs of different users within the renewable energy sector including their decision-making contexts and capacities (Vaughan et al., 2016). Thus the growing technical capacity has facilitated the production of credible and legitimate information, while bringing together researchers and potential users from the start of ECEM has helped to build trust and to ensure that the tailored information produced is also salient (Vaughan et al., 2016; Vaughan and Dessai, 2014). With regular communication including face-to-face during workshops/symposia/tailored meetings, open discussion and willingness to adapt the Demonstrator and other outputs to reflect emerging user needs, it can be argued that ECEM has contributed to the emergence of a community of practice (Vincent et al., 2018). This community of practice is well placed to contribute to the ongoing development of the climate services market (see Section 5). It is inter-disciplinary in that it includes experts in climate, energy, observations, seasonal forecasts and climate projections as well as communications and end users. It could, however, be extended to include, in particular, more expertise in social and decision-making science.

Whilst concluding that ECEM was effective in stakeholder engagement, it is nonetheless acknowledged that more energy industry stakeholders could have been involved in testing the Demonstrator during the pre-operational phase (Fig. 2) and that climate services are still not widely accepted or used in the renewable energy sector. It is also possible that the generally very positive responses to the user survey discussed above reflect, in part, disengagement of any dissatisfied users. Deeper engagement and capacity building could have been achieved through mechanisms such as secondments or internships, either within the consortium itself, or of team members to external organisations and vice versa.

The relatively short timeframe for the development and launch of the pre-operational service (Fig. 2) restricted the amount of time available for quality control of data and development of supporting documentation, particularly case studies, key messages and journal papers, before data were made public. Not all user needs expressed could be met within the constraints of the proof-of-concept, and indeed not all user requests are necessarily feasible at all given the current state of climate science and the availability and reliability of both observed
and simulated data. In such cases, the reasons were always explained to users and these needs were documented for the future (see Section 5). These include requests for higher temporal and spatial resolution, additional variables such as offshore wind and streamflow, and additional Demonstrator functionality such as trend and variability analysis.

## 5. Concluding remarks

Through a carefully planned process of iterative engagement with stakeholders, ECEM successfully developed and launched a pre-operational climate service for the European renewable energy sector, i.e., the C3S ECEM Demonstrator (http://ecem.wemcouncil.org/). This is an interactive visual on-line tool which allows users to view and explore energy supply and demand profiles, and climate variables, for each European country and subnational clusters, in map and time series format (Troccoli et al., 2018) (see also Table 3). ECEM partners and external stakeholders from six targeted categories within this sector (Table 1) worked together (Fig. 2) to develop and evaluate (Table 2) the Demonstrator and to build capacity on both sides. Alongside the tangible outcome of the Demonstrator and its embedded data, thus raising the TRL, the ECEM activities have supported the emergence of a community of practice (Fig. 1).

Members of this community of practice are now recombining in a number of different groupings to work on new initiatives, building on the ECEM and CLIM4ENERGY expertise. These include Horizon2020 research and innovation projects such as SECLI-FIRM (http://www.secli-firm.eu/) and S2S4E (https://s2s4e.eu/) which are assessing the value of seasonal forecasts for the energy sector, MEDSCOPE (https://www.medscope-project.eu/) which is focused on the Mediterranean, and Plan4Res (https://www.plan4res.eu/) which is using the ECEM historical and projections data to assess the impacts of climate change on EU energy systems. Having developed the proof-of-concept, selected partners from ECEM and CLIM4ENERGY have since come together in C3S Energy to work towards delivery of an operational service (https://climate.copernicus.eu/operational-service-energy-sector), which will be fully and transparently integrated with the Copernicus Climate Data Store (https://cds.climate.copernicus.eu). Reflecting this next step towards operationalisation, C3S Energy is co-ordinated through the World Energy and Meteorology Council (see Section 2) rather than an academic institution. It is expected that C3S Energy will raise the TRL to 7 (system prototype demonstration in an operational environment) or 8 (system complete and qualified) (Fig. 1). The C3S-funded operational service will provide a free and public service, i.e. it will be for the 'public good'. It will still leave considerable scope for service companies from both the private (e.g. consultancies) and public (e.g. National Meteorological and Hydrological Services) sectors to use C3S Energy outputs for the development of commercial products and services. Such commercialisation would raise the TRL to 9 (actual system proven in operational environment).

In terms of the challenges underpinning the European Roadmap for Climate Services (E.C., 2015; Street, 2016) it is easier to argue that ECEM has enhanced the quality and relevance of climate services for the energy sector than to assess how it may have enabled market growth or helped build the market framework. Research on the market potential for climate services, such as undertaken in the EU Horizon 2020 EU-MACS (http://eu-macs.eu/) and MARCO (http://marco-h2020.eu/) projects, is still in its infancy (Hoa et al., 2018). When the ECEM proposal was written and submitted, however, it was the case that there were few potential users actively or openly expressing a need for such a service. During its course, ECEM has gradually engaged more and more of the energy community including potential service providers (see Table 1b), making them aware of climate services and information, and thus driving demand for further services.

## Funding

This work on the European Climatic Energy Mixes (ECEM) contract was supported by the Copernicus Climate Change Service (C3S), a programme being implemented by the European Centre for Medium-Range Weather Forecasts (ECMWF) on behalf of the European Commission (grant number: 2015/C3S_441_Lot2_UEA).

## Declaration of Competing Interest

The authors declare that they have no known competing financial interests or personal relationships that could have appeared to influence the work reported in this paper.

## References

Beier, P., Hansen, L.J., Helbrecht, L., Behar, D., 2017. A how-to guide for coproduction of actionable science. Conserv. Lett. 10, 288-296.

Bett, P., Thornton, H., Troccoli, A., 2018a. Skill Assessment of Energy-relevant Climate Variables in a Selection of Seasonal Forecast Models. Report using final data sets. ECEM Deliverable D2.2.1 v2, doi:10.5281/zenodo.1293862.

Bett, P., Thornton, H., de Felice, M., Suckling, E., Dubus, L., Saint-Drenan, Y.-M., Troccoli, A., Goodess, C., 2018b. Assessment of Seasonal Forecasting Skill for Energy Variables. ECEM Deliverable D3.4.1, doi:10.5281/zenodo.1295517.

Brayshaw, D.J., Troccoli, A., Fordham, R., Methven, J., 2011. The impact of large scale atmospheric circulation patterns on wind power generation and its potential predictability: a case study over the UK. Renewable Energy 36, 2087-2096.

Bruno Soares, M., Alexander, M., Dessai, S., 2018. Sectoral use of climate information in Europe: a synoptic overview. Clim. Serv. 9, 5-20.

Buontempo, C., Hewitt, C., 2018. EUPORIAS and the development of climate services. Clim. Serv. 9, 1-4.

Cannon, D.J., Brayshaw, D.J., Methven, J., Coker, P.J., Lenaghan, D., 2015. Using re-analysis data to quantify extreme wind power generation statistics: a 33 year case study in Great Britain. Renewable Energy 75, 767-778.

Clark, R.T., Bett, P.E., Thornton, H.E., Scaife, A.A., 2017. Skillful seasonal predictions for the European energy industry. Environ. Res. Lett. 12 024002.

Damm, A., Köberl, J., Prettenthaler, F., Rogler, N., Töglhofer, C., 2017. Impacts of +2°C global warming on electricity demand in Europe. Clim. Serv. 7, 12-30.

De Felice, M., Dubus, L., Suckling, E., Troccoli, A., 2018. The impact of the North Atlantic Oscillation on European hydro-power generation. EarthArXiv preprint. doi:10.31223/osf.io/8sntx.

Doblas-Reyes, F.J., Garcia-Serrano, J., Lienert, F., Biescas, A.P., Rodrigues, L.R.L., 2013. Seasonal climate predictability and forecasting: status and prospects. WIRES Clim. Change. 4, 245-268.

Dubus, L., Claudel, S., De Felice, M., Saint-Drenan, Y.-M., Troccoli, A., Goodess, C.M., 2019. A C3S comprehensive dataset of power demand and supply for different climatic time scales over Europe. In preparation.

Durose, C., Richardson, L., Perry, B., 2018. Craft metrics to value co-production. Nature 562, 32-33.

E.C. (European Commission), 2015. A European Research and Innovation Roadmap for Climate 2015 Services Brussels. < https://ec.europa.eu/programmes/horizon2020/en/news/european-research-and-innovation-roadmap-climate-services > (accessed 13.12.18).

Ebinger, J., Vergara, W. (Eds.), 2011. Climate Impacts on Energy Systems: Key Issues for Energy Sector Adaptation. World Bank publication. < http://documents.worldbank.org/curated/en/580481468331850839/Climate-impacts-on-energy-systems-key-issues-for-energy-sector-adaptation > (accessed 13.12.18).

Gardiner, E.P., Herring, D.D., Fox, J.F., 2018. The U.S. Climate Resilience Toolkit: evidence of progress. Clim. Change. https://doi.org/10.1007/s10584-018-2216-0.

Haße, C., Kind, C., 2018. Updating an existing online adaptation support tool: insights from an evaluation. Clim. Change. https://doi.org/10.1007/s10584-018-2166-6.

Heder, M., 2017. From NASA to EU: the evolution of the TRL scale in public sector innovation. Innovation J. 22, 23.

Hewitt, C.D., Stone, R.C., Tait, A.B., 2017. Improving the use of climate information in decision-making. Nat. Clim. Change 7, 614-616.

Hoa, E., Perrels, A., Le, T.-T., 2018. From generating to using climate services – how the EU-MACS and MARCO projects help to unlock the market potential. Clim. Serv. 11, 86-88.

Jerez, S., Tobin, I., Vautard, R., Montavez, J.P., Lopez-Romero, J.M., Thais, F., Bartok, B., Christensen, O.B., Colette, A., Deque, M., Nikulin, G., Kotlarski, S., van Meijgaard, E., Teichmann, C., Wild, M., 2015. The impact of climate change on photovoltaic power generation in Europe. Nat. Commun. 6, 10014.

Johnston, P.C., Gomez, J.F., Laplante, B., 2012. Climate Risk and Adaptation in the Electric Power Sector. Asian Development Bank publication. Available at: < https://www.adb.org/sites/default/files/publication/29889/climate-risks-adaptation-power-sector.pdf > (accessed 13.12.18).

Jones, P.D., Harpham, C., Troccoli, A., Gschwind, B., Ranchin, T., Wald, L., Goodess, C.M., Dorling, S., 2017. Using ERA-Interim reanalysis for creating datasets of energy-relevant climate variables. Earth Syst. Sci. Data 9, 471-495.

Laudien, R., Boon, E., Goosen, H., van Nieuwaal, K., 2018. The Dutch adaptation web portal: seven lessons learnt from a co-production point of view. Clim. Change.
https://doi.org/10.1007/s10584-018-2179-1.

Lemos, M.C., Kirchhoff, C.J., Ramprasad, V., 2012. Narrowing the climate information usability gap. Nat. Clim. Change 2, 789–794.

McNie, E.C., 2013. Delivering climate services: organizational strategies and approaches for producing useful climate-science information. Weather Clim. Soc. 5, 14–26.

Meadow, A.M., Ferguson, D.B., Guido, Z., Horangic, A., Owen, G., Wall, T., 2015. Moving toward the deliberate coproduction of climate science knowledge. Weather Clim. Soc. 7, 179–191.

Palin, E.J., Scaife, A.A., Wallace, E., Pope, E.C.D., Arribas, A., Brookshaw, A., 2016. Skillful seasonal forecasts of winter disruption to the U.K. transport system. J. Appl. Meteorol. Climatol. 55, 325–344.

Saint-Drenan, Y.M., Wald, L., Ranchin, T., Dubus, L., Troccoli, A., 2018. An approach for the estimation of the aggregated photovoltaic power generated in several European countries from meteorological data. Adv. Sci. Res. 147, 257–276.

Schaeffer, R., Szklo, A.S., de Lucena, A.F.P., Borba, B.S.M.C., Nogueira, L.P.P., Fleming, F.P., Troccoli, A., Harrison, M., Boulahya, M.S., 2012. Energy sector vulnerability to climate change: a review. Energy. 38, 1–12.

Schuck-Zöller, S., Cortekar, J., Jacob, D., 2017. Evaluating co-creation of knowledge: from quality criteria and indicators to methods. Adv. Sci. Res. 14, 305–312.

Soares, M.B., Dessai, S., 2015. Exploring the use of seasonal climate forecasts in Europe through expert elicitation. Clim. Risk Manage. 10, 8–16.

Soares, M.B., Dessai, S., 2016. Barriers and enablers to the use of seasonal climate forecasts amongst organisations in Europe. Clim. Change 137, 89–103.

Street, R.B., 2016. Towards a leading role on climate services in Europe: a research and innovation roadmap. Clim. Serv. 1, 2–5.

Swart, R.J., de Bruin, K., Dhenain, S., Dubois, G., Groot, A., von der Forst, E., 2017. Developing climate information portals with users: promises and pitfalls. Clim. Serv. 6, 12–22.

Taylor, A., Scott, D., Steynor, A., McClure, A., 2017. Transdisciplinarity, Co-production and Coexploration: Integrating Knowledge Across Science, Policy and Practice in FRACTAL. FRACTAL working paper #3, 19 pp. < http://www.fractal.org.za/wp-content/uploads/2017/03/Co-co-trans_March-2017.pdf > (accessed 13.12.18).

Thornton, H.E., Scaife, A.A., Hoskins, B.J., Brayshaw, D.J., 2017. The relationship between wind power, electricity demand and winter weather patterns in Great Britain. Environ. Res. Lett. 12 064017.

Tobin, I., Vautard, R., Balog, I., Breon, F.M., Jerez, S., Ruti, P.M., Thais, F., Vrac, M., Yiou, P., 2015. Assessing climate change impacts on European wind energy from ENSEMBLES high-resolution climate projections. Clim. Change 128, 99–112.

Troccoli, A., Goodess, C., Jones, P., Penny, L., Dorling, S., Harpham, C., Dubus, L., Parey, S., Claudel, S., Khong, D.-H., Bett, P.E., Thornton, H., Ranchin, T., Wald, L., Saint-Drenan, Y.-M., De Felice, M., Brayshaw, D., Suckling, E., Percy, B., Blower, J., 2018. Creating a proof-of-concept climate service to assess future renewable energy mixes in Europe: An overview of the C3S ECEM project. Adv. Sci. Res. 15, 191–205.

Troccoli, A. (Ed.), 2018. Weather and Climate Services for the Energy Industry. Palgrave Macmillan, Cham, Switzerland, 212 pp. doi:10.1007/978-3-319-68418-5 (accessed 13.12.18).

van Vliet, M.T.H., Wiberg, D., Leduc, S., Riahi, K., 2016. Power-generation system vulnerability and adaptation to changes in climate and water resources. Nat. Clim. Change 6, 375–380.

van Vuuren, D.P., Edmonds, J., Kainuma, M., Riahi, K., Thomson, A., Hibbard, K., Hurtt, G.C., Kram, T., Krey, V., Lamarque, J.-F., Masui, T., Meinshausen, M., Nakicenovic, N., Smith, S.J., Rose, S.K., 2011. The representative concentration pathways: an overview. Clim. Change 109, 5–31.

Vaughan, C., Dessai, S., 2014. Climate services for society: origins, institutional arrangements, and design elements for an evaluation framework. WIRES Clim. Change 5, 587–603.

Vaughan, C., Buja, L., Kruczkiewicz, A., Goddard, L., 2016. Identifying research priorities to advance climate services. Clim. Serv. 4, 65–74.

Vincent, K., Steynor, A., Waagsaether, K., Cull, T., 2018. Communities of practice: one size does not fit all. Clim. Serv. 11, 72–77.

Wenz, L., Levermann, A., Auffhammer, M., 2017. North-south polarization of European electricity consumption under future warming. Proc. Nat. Acad. Sci. 114, E7910–E7918.

Willyard, C., Scudellari, M., Nordling, L., 2018. Partners in science - the people who should benefit from research are increasingly shaping how it is done. Nature 562, 24–28.

WMO (World Meteorological Organization), 2017. Energy Exemplar to the User Interface Platform of the Global Framework for Climate Services. < https://gfcs.wmo.int/sites/default/files/Priority-Areas/Energy/GFCS_Energy Exemplar_JN17453.pdf > (accessed 13.12.18).