Repository: electricitymaps/electricitymaps-contrib

Issue:
Missing oil generation in Ibiza and Menorca
## Description
It seems the values for diesel (oil) generation are not being recorded:


![Screenshot_20230712_164941](https://github.com/electricitymaps/electricitymaps-contrib/assets/84802276/2e4d2aea-81be-4b5d-9a91-b29c6e451b35)
![Screenshot_20230712_165018](https://github.com/electricitymaps/electricitymaps-contrib/assets/84802276/a91ceb76-662b-4f86-8c63-efa283c0fa83)

This happens also in Menorca.

Running the parser locally shows oil production as 0 MW:
```bash
{'correctedModes': [],
  'datetime': datetime.datetime(2023, 7, 12, 17, 0, tzinfo=datetime.timezone.utc),
  'production': {'biomass': 0.0,
                 'coal': 0.0,
                 'gas': 82.4,
                 'hydro': 0.0,
                 'nuclear': 0.0,
                 'oil': 0.0,
                 'solar': 1.5,
                 'unknown': 0.0,
                 'wind': 0.0},
  'source': 'demanda.ree.es',
  'sourceType': <EventSourceType.measured: 'measured'>,
  'storage': {},
  'zoneKey': 'ES-IB-IZ'}
```


Discussion:
It seems like the package we use to interface with the API is not picking this up. I'll look into it and see if I can submit an update.
So I kind of took this one step further and upgraded large parts of ree as well but the PR is open here at least: https://github.com/hectorespert/ree/pull/13
Slightly off-topic, but are we sure the 'gas turbines' are actually burning natural gas? A lot of gas turbine plants are actually running on oil-based fuel, so I'm bringing it up in case someone saw 'gas' and thought 'natural gas' when creating this parser.
While I'm not 100% sure, there is a gas pipeline that goes to the Balearic islands so at least there is the infrastructure for it.

We are however overriding that for the canary islands as the LNG terminals have not been built yet (last I checked) and treating it as oil. 
After searching a bit, I've found that the only islands using oil are Menorca and Formentera. Ibiza power plant was changed to natural gas, so both "Diesel engines" and "Gas turbine" use natural gas now. In Menorca both use diesel.
That complicates things significantly.
Because ree (the package) combines both auxiliary generation and diesel to oil which means there is no way for us to split it and I would think the auxilary generation (backups) still use diesel oil?

Might be easier for us to just parse the sources directly and than using the REE package, or we would have to make even more changes to it (which is a little outside of our control).
Um, but wasn't auxiliary generation only used in Formentera? If this is the case, the fact that they are added doesn't cause any problems, we'll just have to parse diesel to natural gas in the case of Ibiza.
GenAux is included for the Ibiza dataset as well although I don't know if they are reporing any values for it as it's zero (for the day I checked).
Here is the endpoint and values that are reported: [demanda.ree.es/WSvisionaMovilesBalearesRest/resources/demandaGeneracionBaleares?curva=IBIZA5M&fecha=2023-07-12](https://demanda.ree.es/WSvisionaMovilesBalearesRest/resources/demandaGeneracionBaleares?curva=IBIZA5M&fecha=2023-07-12)
In case a rewrite of ree or the parser is needed, I'd like to point out that diesel and gas turbines seem to have different carbon intensities, as can be seen in demanda.ree.es. Could we make the parser so that this is taken into account? I've seen that in many cases you just use precalculated averages based on the previous years production and aggregate different production types, but I feel this isn't as accurate, and doesn't show why/if one production type is favoured to decrease CO2 emissions. Is this a design limitation or is the parser feasible without changes to the codebase? Sorry if I went a bit off-topic.
That's fine and yes we do aggregate some data. I wouldn't say it limitation rather than a design choice to better unify all the parser data for all zones so we can say this is coal, even if one source reports all coal as coal or another breaks it down by type. With that said I have suggested the same thing here https://github.com/electricitymaps/electricitymaps-contrib/discussions/4219#discussioncomment-3110126 before I started working at Electricity Maps.

With the suggested changes in there we could compute hourly (or better) carbon intensities based on the raw production and still aggregate it to the current modes we use now and use the weighted average emission factor instead.

I'll see if I can push for this a little more internally but as you said that is off topic for this particular issue.
If there is no activity from the ree maintainer in next few days I think we should go ahead and create our own parser, if nothing else so we have control over the source code and any changes in there.
Any news on this? I can work on a new parser.
> Any news on this? I can work on a new parser.

No new news. I just haven't had the time.

The old parser works for the most part. We just need to replace the usage of REE with our own data fetching so we can assign the raw data as we please.

If you want to take a look at it feel free to do so.

Resolving patch:
diff --git a/parsers/ES.py b/parsers/ES.py
--- a/parsers/ES.py
+++ b/parsers/ES.py
@@ -1,27 +1,9 @@
 #!/usr/bin/env python3
 
-from collections.abc import Callable
-from datetime import datetime, timedelta, timezone
+from datetime import datetime, timedelta
+from json import loads
 from logging import Logger, getLogger
-
-# package "ree" is used to parse data from www.ree.es
-# maintained on github by @hectorespert at https://github.com/hectorespert/ree
-from ree import (
-    Ceuta,
-    ElHierro,
-    Formentera,
-    Gomera,
-    GranCanaria,
-    IberianPeninsula,
-    Ibiza,
-    LanzaroteFuerteventura,
-    LaPalma,
-    Mallorca,
-    Melilla,
-    Menorca,
-    Response,
-    Tenerife,
-)
+from zoneinfo import ZoneInfo
 
 # The request library is used to fetch content through HTTP
 from requests import Session
@@ -37,45 +19,127 @@
 from .lib.config import refetch_frequency
 from .lib.exceptions import ParserException
 
-SOURCE = "demanda.ree.es"
-
+#    Zone name Cheat Sheet
+
+#    ZoneKey("ES"): "IberianPeninsula"
+#    ZoneKey("ES-CE"): "Ceuta"
+#    ZoneKey("ES-CN-FVLZ"): "LanzaroteFuerteventura"
+#    ZoneKey("ES-CN-GC"): "GranCanaria"
+#    ZoneKey("ES-CN-HI"): "ElHierro"
+#    ZoneKey("ES-CN-IG"): "Gomera"
+#    ZoneKey("ES-CN-LP"): "LaPalma"
+#    ZoneKey("ES-CN-TE"): "Tenerife"
+#    ZoneKey("ES-IB-FO"): "Formentera"
+#    ZoneKey("ES-IB-IZ"): "Ibiza"
+#    ZoneKey("ES-IB-MA"): "Mallorca"
+#    ZoneKey("ES-IB-ME"): "Menorca"
+#    ZoneKey("ES-ML"): "Melilla"
+
+PRODUCTION_PARSE_MAPPING = {
+    "nuc": "nuclear",
+    "die": "oil",
+    "genAux": "oil",
+    "gas": "gas",
+    "gf": "gas",
+    "eol": "wind",
+    "cc": "gas",
+    "vap": "oil",
+    "fot": "solar",
+    "sol": "solar",
+    "car": "coal",
+    "resid": "biomass",
+    "termRenov": "unknown",
+    "cogenResto": "unknown",
+    "tnr": "unknown",
+    "trn": "unknown",
+    "otrRen": "unknown",
+    "cogen": "unknown",
+}
 
-ZONE_FUNCTION_MAP: dict[ZoneKey, Callable] = {
-    ZoneKey("ES"): IberianPeninsula,
-    ZoneKey("ES-CE"): Ceuta,
-    ZoneKey("ES-CN-FVLZ"): LanzaroteFuerteventura,
-    ZoneKey("ES-CN-GC"): GranCanaria,
-    ZoneKey("ES-CN-HI"): ElHierro,
-    ZoneKey("ES-CN-IG"): Gomera,
-    ZoneKey("ES-CN-LP"): LaPalma,
-    ZoneKey("ES-CN-TE"): Tenerife,
-    ZoneKey("ES-IB-FO"): Formentera,
-    ZoneKey("ES-IB-IZ"): Ibiza,
-    ZoneKey("ES-IB-MA"): Mallorca,
-    ZoneKey("ES-IB-ME"): Menorca,
-    ZoneKey("ES-ML"): Melilla,
+EXCHANGE_PARSE_MAPPING = {
+    "cb": "pe_ma",
+    "icb": "pe_ma",
+    "emm": "ma_me",
+    "emi": "ma_ib",
+    "eif": "ib_fo",
+    "inter": "int",
 }
 
-EXCHANGE_FUNCTION_MAP: dict[str, Callable] = {
-    "ES->ES-IB-MA": Mallorca,
-    "ES-IB-IZ->ES-IB-MA": Mallorca,
-    "ES-IB-FO->ES-IB-IZ": Formentera,
-    "ES-IB-MA->ES-IB-ME": Mallorca,
+ZONE_MAPPING = {
+    ZoneKey("ES"): {
+        "API_CODE": "DEMANDAQH",
+        "SYSTEM": "Peninsula",
+        "TZ": "Europe/Madrid",
+    },
+    ZoneKey("ES-CE"): {
+        "API_CODE": "CEUTA5M",
+        "SYSTEM": "Peninsula",
+        "TZ": "Africa/Ceuta",
+    },
+    ZoneKey("ES-CN-FVLZ"): {
+        "API_CODE": "LZ_FV5M",
+        "SYSTEM": "Canarias",
+        "TZ": "Atlantic/Canary",
+    },
+    ZoneKey("ES-CN-GC"): {
+        "API_CODE": "GCANARIA5M",
+        "SYSTEM": "Canarias",
+        "TZ": "Atlantic/Canary",
+    },
+    ZoneKey("ES-CN-HI"): {
+        "API_CODE": "EL_HIERRO5M",
+        "SYSTEM": "Canarias",
+        "TZ": "Atlantic/Canary",
+    },
+    ZoneKey("ES-CN-IG"): {
+        "API_CODE": "LA_GOMERA5M",
+        "SYSTEM": "Canarias",
+        "TZ": "Atlantic/Canary",
+    },
+    ZoneKey("ES-CN-LP"): {
+        "API_CODE": "LA_PALMA5M",
+        "SYSTEM": "Canarias",
+        "TZ": "Atlantic/Canary",
+    },
+    ZoneKey("ES-CN-TE"): {
+        "API_CODE": "TENERIFE5M",
+        "SYSTEM": "Canarias",
+        "TZ": "Atlantic/Canary",
+    },
+    ZoneKey("ES-IB-FO"): {
+        "API_CODE": "FORMENTERA5M",
+        "SYSTEM": "Baleares",
+        "TZ": "Europe/Madrid",
+    },
+    ZoneKey("ES-IB-IZ"): {
+        "API_CODE": "IBIZA5M",
+        "SYSTEM": "Baleares",
+        "TZ": "Europe/Madrid",
+    },
+    ZoneKey("ES-IB-MA"): {
+        "API_CODE": "MALLORCA5M",
+        "SYSTEM": "Baleares",
+        "TZ": "Europe/Madrid",
+    },
+    ZoneKey("ES-IB-ME"): {
+        "API_CODE": "MENORCA5M",
+        "SYSTEM": "Baleares",
+        "TZ": "Europe/Madrid",
+    },
+    ZoneKey("ES-ML"): {
+        "API_CODE": "MELILLA5M",
+        "SYSTEM": "Peninsula",
+        "TZ": "Africa/Ceuta",
+    },
 }
 
-# dict of REE production types to Electricity Maps production types
-PRODUCTION_MAPPING = {
-    "carbon": "coal",
-    "combined": "gas",
-    "diesel": "oil",
-    "gas": "gas",
-    "hydraulic": "hydro",
-    "nuclear": "nuclear",
-    "other": "unknown",
-    "solar": "solar",
-    "vapor": "oil",
-    "waste": "biomass",
-    "wind": "wind",
+SOURCE = "demanda.ree.es"
+
+EXCHANGE_FUNCTION_MAP: dict[ZoneKey, ZoneKey] = {
+    ZoneKey("ES->ES-IB-MA"): ZoneKey("ES-IB-MA"),
+    ZoneKey("ES-IB-IZ->ES-IB-MA"): ZoneKey("ES-IB-MA"),
+    ZoneKey("ES-IB-FO->ES-IB-IZ"): ZoneKey("ES-IB-FO"),
+    ZoneKey("ES-IB-MA->ES-IB-ME"): ZoneKey("ES-IB-MA"),
 }
 
 
@@ -85,7 +149,7 @@ def check_valid_parameters(
     target_datetime: datetime | None,
 ):
     """Raise an exception if the parameters are not valid for this parser."""
-    if "->" not in zone_key and zone_key not in ZONE_FUNCTION_MAP.keys():
+    if "->" not in zone_key and zone_key not in ZONE_MAPPING.keys():
         raise ParserException(
             "ES.py",
             f"This parser cannot parse data for zone: {zone_key}",
@@ -112,16 +176,53 @@ def check_valid_parameters(
         )
 
 
-def fetch_island_data(
-    zone_key: ZoneKey, session: Session, target_datetime: datetime | None
-) -> list[Response]:
-    """Fetch data for the given zone key."""
+def check_known_key(key: str, logger: Logger):
+    """Check if the given key is already known and log a warning if not."""
+    if (
+        key not in EXCHANGE_PARSE_MAPPING
+        and key not in PRODUCTION_PARSE_MAPPING
+        and key not in {"dem", "hid", "ts"}
+    ):
+        logger.warning(f'Key "{key}" could not be parsed!')
+
+
+def get_ree_data(
+    zone_key: ZoneKey, session: Session, target_datetime: datetime | None, tz: str
+) -> dict:
     if target_datetime is None:
-        date = target_datetime
+        date = datetime.now(tz=ZoneInfo(tz)).strftime("%Y-%m-%d")
     else:
         date = target_datetime.strftime("%Y-%m-%d")
+    system = ZONE_MAPPING[zone_key]["SYSTEM"]
+    zone = ZONE_MAPPING[zone_key]["API_CODE"]
+    res = session.get(
+        f"https://demanda.ree.es/WSvisionaMoviles{system}Rest/resources/demandaGeneracion{system}?curva={zone}&fecha={date}"
+    )
+    if not res.ok:
+        raise ParserException(
+            "ES.py",
+            f"Failed fetching data for {zone_key}",
+            zone_key,
+        )
+    json = loads(res.text.replace("null(", "", 1).replace(r");", "", 1))
+    return json["valoresHorariosGeneracion"]
+
 
-    data: list[Response] = ZONE_FUNCTION_MAP[zone_key](session).get_all(date)
+def fetch_and_preprocess_data(
+    zone_key: ZoneKey,
+    session: Session,
+    logger: Logger,
+    target_datetime: datetime | None,
+):
+    """Fetch data for the given zone key."""
+    tz = ZONE_MAPPING[zone_key]["TZ"]
+    data = get_ree_data(zone_key, session, target_datetime, tz)
+    for value in data:
+        # Add timezone info to time object
+        value["ts"] = datetime.fromisoformat(value["ts"]).replace(tzinfo=ZoneInfo(tz))
+
+        for key in value.keys():
+            check_known_key(key, logger)
     if data:
         return data
     else:
@@ -142,13 +243,13 @@ def fetch_consumption(
     check_valid_parameters(zone_key, session, target_datetime)
 
     ses = session or Session()
-    island_data = fetch_island_data(zone_key, ses, target_datetime)
+    data = fetch_and_preprocess_data(zone_key, ses, logger, target_datetime)
     consumption = TotalConsumptionList(logger)
-    for event in island_data:
+    for event in data:
         consumption.append(
             zoneKey=zone_key,
-            datetime=datetime.fromtimestamp(event.timestamp).astimezone(timezone.utc),
-            consumption=event.demand,
+            datetime=event["ts"],
+            consumption=event["dem"],
             source="demanda.ree.es",
         )
     return consumption.to_list()
@@ -163,40 +264,37 @@ def fetch_production(
 ) -> list[dict]:
     check_valid_parameters(zone_key, session, target_datetime)
     ses = session or Session()
-    island_data = fetch_island_data(zone_key, ses, target_datetime)
+    data_mapping = PRODUCTION_PARSE_MAPPING.copy()
+
+    ## Production mapping override for Canary Islands
+    # NOTE the LNG terminals are not built yet, so power generated by "gas" or "cc" in ES-CN domain is actually using oil.
+    # Recheck this every 6 months and move to gas key if there has been a change.
+    # Last checked: 2022-06-27
+    if zone_key.split("-")[1] == "CN":
+        data_mapping["gas"] = "oil"
+        data_mapping["cc"] = "oil"
+
+    if zone_key == "ES-IB-ME" or zone_key == "ES-IB-FO":
+        data_mapping["gas"] = "oil"
+    if zone_key == "ES-IB-IZ":
+        data_mapping["die"] = "gas"
+
+    data = fetch_and_preprocess_data(zone_key, ses, logger, target_datetime)
     productionEventList = ProductionBreakdownList(logger)
+    for event in data:
 
-    for event in island_data:
-        production = ProductionMix()
         storage = StorageMix()
+        if "hid" in event:
+            storage.add_value("hydro", -event["hid"])
 
-        for mode in PRODUCTION_MAPPING.keys():
-            if mode in event.__dir__():
-                value = getattr(event, mode)
-                ## Production mapping override for Canary Islands
-                # NOTE the LNG terminals are not built yet, so power generated by "gas" or "combined" in ES-CN domain is actually using oil.
-                # Recheck this every 6 months and move to gas key if there has been a change.
-                # Last checked: 2022-06-27
-                fuel_mapping = PRODUCTION_MAPPING
-                if zone_key.split("-")[1] == "CN":
-                    fuel_mapping = {
-                        **PRODUCTION_MAPPING,
-                        "gas": "oil",
-                        "combined": "oil",
-                    }
-                    # Reset gas to 0, as it is now oil, normally this should be skipped or set to None but to conform to how REE does it, we set it to 0
-                    # We should aim to change this upstream in the future to make it uniform with other parsers going forward.
-                    production.gas = 0
-                production.add_value(fuel_mapping[mode], value, True)
-                # Zone specific override for El Hierro
-                # Hydro response is hydro storage, not hydro production
-                if zone_key == "ES-CN-HI" and mode == "hydraulic":
-                    storage.hydro = -value
-                    production.hydro = 0
+        production = ProductionMix()
+        for key in event:
+            if key in data_mapping.keys():
+                production.add_value(data_mapping[key], event[key])
 
         productionEventList.append(
             zoneKey=zone_key,
-            datetime=datetime.fromtimestamp(event.timestamp).astimezone(timezone.utc),
+            datetime=event["ts"],
             production=production,
             storage=storage,
             source="demanda.ree.es",
@@ -215,35 +313,35 @@ def fetch_exchange(
 ) -> list[dict]:
     sorted_zone_keys = ZoneKey("->".join(sorted([zone_key1, zone_key2])))
     check_valid_parameters(sorted_zone_keys, session, target_datetime)
-
-    if isinstance(target_datetime, datetime):
-        date = target_datetime.strftime("%Y-%m-%d")
-    else:
-        date = target_datetime
-
     ses = session or Session()
 
-    responses: list[Response] = EXCHANGE_FUNCTION_MAP[sorted_zone_keys](ses).get_all(
-        date
+    data = fetch_and_preprocess_data(
+        EXCHANGE_FUNCTION_MAP[sorted_zone_keys],
+        ses,
+        logger,
+        target_datetime,
     )
 
     exchangeList = ExchangeList(logger)
-    for response in responses:
+    for event in data:
+        exchanges = {}
+        for key in event:
+            if key in EXCHANGE_PARSE_MAPPING.keys():
+                exchanges[EXCHANGE_PARSE_MAPPING[key]] = event[key]
+
         net_flow: float
         if sorted_zone_keys == "ES-IB-MA->ES-IB-ME":
-            net_flow = -1 * response.link["ma_me"]
+            net_flow = -1 * exchanges["ma_me"]
         elif sorted_zone_keys == "ES-IB-IZ->ES-IB-MA":
-            net_flow = response.link["ma_ib"]
+            net_flow = exchanges["ma_ib"]
         elif sorted_zone_keys == "ES-IB-FO->ES-IB-IZ":
-            net_flow = -1 * response.link["ib_fo"]
+            net_flow = -1 * exchanges["ib_fo"]
         else:
-            net_flow = response.link["pe_ma"]
+            net_flow = exchanges["pe_ma"]
 
         exchangeList.append(
             zoneKey=sorted_zone_keys,
-            datetime=datetime.fromtimestamp(response.timestamp).astimezone(
-                timezone.utc
-            ),
+            datetime=event["ts"],
             netFlow=net_flow,
             source="demanda.ree.es",
         )


Test patch:
diff --git a/parsers/test/test_ES.py b/parsers/test/test_ES.py
--- a/parsers/test/test_ES.py
+++ b/parsers/test/test_ES.py
@@ -2,7 +2,6 @@
 from unittest import TestCase, main
 from unittest.mock import patch
 
-from ree import ElHierro, Formentera, Mallorca, Response
 from requests import Session
 
 from electricitymap.contrib.lib.types import ZoneKey
@@ -13,137 +12,143 @@ class TestES(TestCase):
     def setUp(self):
         self.session = Session()
 
-        first_request = Response(1687192328)
-        first_request.link["pe_ma"] = 50
-        first_request.link["ib_fo"] = 10
+    ### El Hierro
+    # Consumption
+    @patch("requests.Response")
+    @patch("parsers.ES.Session.get")
+    def test_fetch_consumption(self, mocked_session_get, mocked_response):
+        mocked_response.ok = True
+        mocked_response.text = r'null({"valoresHorariosGeneracion":[{"ts":"2023-09-04 00:55","dem":5.5,"die":3.2,"gas":0.0,"eol":3.0,"cc":0.0,"vap":0.0,"fot":0.0,"hid":-0.5}]}'
+        mocked_session_get.return_value = mocked_response
+        data_list = ES.fetch_consumption(
+            ZoneKey("ES-CN-HI"), self.session, datetime.fromisoformat("2023-09-04")
+        )
 
-        second_request = Response(1687192357)
-        second_request.link["pe_ma"] = 50
-        second_request.link["ib_fo"] = 10
+        # Test "get" function has been called correctly
+        self.assertEqual(
+            mocked_session_get.call_args[0][0],
+            "https://demanda.ree.es/WSvisionaMovilesCanariasRest/resources/demandaGeneracionCanarias?curva=EL_HIERRO5M&fecha=2023-09-04",
+        )
 
-        self.mocked_responses = [first_request, second_request]
+        # Test that the data is parsed correctly afterwards
+        self.assertEqual(len(data_list), 1)
+        self.assertEqual(data_list[0]["zoneKey"], "ES-CN-HI")
+        self.assertEqual(data_list[0]["source"], "demanda.ree.es")
+        self.assertEqual(data_list[0]["consumption"], 5.5)
+        self.assertTrue(isinstance(data_list[0]["datetime"], datetime))
+        self.assertEqual(
+            data_list[0]["datetime"],
+            datetime(2023, 9, 3, 23, 55, tzinfo=timezone.utc),
+        )
 
     ### El Hierro
-    # Consumption
-    @patch.object(ElHierro, "get_all")
-    def test_fetch_consumption(self, mocked_get_all):
-        mocked_get_all.return_value = self.mocked_responses
-        data_list = ES.fetch_consumption(ZoneKey("ES-CN-HI"), self.session)
+    # Production
+    @patch("requests.Response")
+    @patch("parsers.ES.Session.get")
+    def test_fetch_production_storage(self, mocked_session_get, mocked_response):
+        mocked_response.ok = True
+        mocked_response.text = r'null({"valoresHorariosGeneracion":[{"ts":"2023-09-04 00:55","dem":5.5,"die":3.2,"gas":0.0,"eol":3.0,"cc":0.0,"vap":0.0,"fot":0.0,"hid":-0.5}]}'
+        mocked_session_get.return_value = mocked_response
+        data_list = ES.fetch_production(
+            ZoneKey("ES-CN-HI"), self.session, datetime.fromisoformat("2023-09-04")
+        )
 
-        self.assertEqual(len(data_list), 0)
+        # Test "get" function has been called correctly
+        self.assertEqual(
+            mocked_session_get.call_args[0][0],
+            "https://demanda.ree.es/WSvisionaMovilesCanariasRest/resources/demandaGeneracionCanarias?curva=EL_HIERRO5M&fecha=2023-09-04",
+        )
 
+        # Test that the data is parsed correctly afterwards
+        self.assertEqual(len(data_list), 1)
+        self.assertEqual(data_list[0]["zoneKey"], "ES-CN-HI")
+        self.assertEqual(data_list[0]["source"], "demanda.ree.es")
+        self.assertEqual(
+            data_list[0]["production"],
+            {
+                "oil": 3.2,
+                "solar": 0.0,
+                "wind": 3.0,
+            },
+        )
+        self.assertEqual(data_list[0]["storage"], {"hydro": 0.5})
+        self.assertTrue(isinstance(data_list[0]["datetime"], datetime))
+        self.assertEqual(
+            data_list[0]["datetime"],
+            datetime(2023, 9, 3, 23, 55, tzinfo=timezone.utc),
+        )
+
+    ### Menorca
     # Production
-    @patch.object(ElHierro, "get_all")
-    def test_fetch_production_el_hierro(self, mocked_get_all):
-        mocked_get_all.return_value = [
-            Response(1687192328, 10.0, 10, 5, 13, 3, 4, 3.5, 10, 5, 0, 3),
-        ]
-        data_list = ES.fetch_production(ZoneKey("ES-CN-HI"), self.session)
-        for data in data_list:
-            self.assertEqual(data["zoneKey"], "ES-CN-HI")
-            self.assertEqual(data["source"], "demanda.ree.es")
-            self.assertTrue(isinstance(data["datetime"], datetime))
-            self.assertIsNotNone(data["production"])
-            self.assertIsNotNone(data["storage"])
-            self.assertEqual(
-                data["datetime"],
-                datetime.utcfromtimestamp(1687192328).replace(tzinfo=timezone.utc),
-            )
-            self.assertEqual(data["production"]["gas"], 0)
-            self.assertEqual(data["production"]["oil"], 22)
-            self.assertEqual(data["production"]["hydro"], 0)
-            self.assertEqual(data["production"]["solar"], 3.5)
-            self.assertEqual(data["production"]["wind"], 13)
-            self.assertEqual(data["production"]["coal"], 5)
-            self.assertEqual(data["production"]["unknown"], 3)
-            self.assertEqual(data["storage"]["hydro"], -10)
-
-    ### Formentera
-    # Production
-    @patch.object(Formentera, "get_all")
-    def test_fetch_production_ES_IB_FO(self, mocked_get_all):
-        mocked_get_all.return_value = [
-            Response(
-                timestamp=1687192328,
-                demand=10.0,
-                diesel=0,
-                gas=5,
-                wind=13,
-                combined=3,
-                vapor=4,
-                solar=3.5,
-                hydraulic=10,
-                carbon=5,
-                other=3,
-            ),
-            Response(
-                timestamp=1687192328,
-                demand=10.0,
-                diesel=-10,  # Test negative value
-                gas=5,
-                wind=13,
-                combined=3,
-                vapor=4,
-                solar=3.5,
-                hydraulic=10,
-                carbon=5,
-                other=3,
-            ),
-        ]
-        data_list = ES.fetch_production(ZoneKey("ES-IB-FO"), self.session)
-        self.assertEqual(len(data_list), 2)
-        for data in data_list:
-            self.assertEqual(data["zoneKey"], "ES-IB-FO")
-            self.assertEqual(data["source"], "demanda.ree.es")
-            self.assertTrue(isinstance(data["datetime"], datetime))
-            self.assertIsNot(data["production"], {})
-            self.assertEqual(
-                data["datetime"],
-                datetime.utcfromtimestamp(1687192328).replace(tzinfo=timezone.utc),
-            )
-            self.assertEqual(data["production"]["gas"], 8)
-            self.assertEqual(data["production"]["oil"], 4)
-            self.assertEqual(data["production"]["hydro"], 10)
-            self.assertEqual(data["production"]["solar"], 3.5)
-            self.assertEqual(data["production"]["wind"], 13)
-            self.assertEqual(data["production"]["coal"], 5)
-            self.assertEqual(data["production"]["unknown"], 3)
+    @patch("requests.Response")
+    @patch("parsers.ES.Session.get")
+    def test_fetch_production(self, mocked_session_get, mocked_response):
+        mocked_response.ok = True
+        mocked_response.text = r'null({"valoresHorariosGeneracion":[{"ts":"2023-09-02 21:00","dem":85.9,"car":0.0,"die":22.6,"gas":59.3,"cc":0.0,"cb":0.0,"fot":1.1,"tnr":0.0,"trn":0.0,"eol":0.2,"emm":2.7,"emi":-0.0,"otrRen":0.0,"resid":0.0,"genAux":0.0,"cogen":0.0,"eif":-0.0}]}'
+        mocked_session_get.return_value = mocked_response
+        data_list = ES.fetch_production(
+            ZoneKey("ES-IB-ME"), self.session, datetime.fromisoformat("2023-09-03")
+        )
 
-    # Exchange
-    @patch.object(Formentera, "get_all")
-    def test_fetch_exchange_ES_IB_FO(self, mocked_get_all):
-        mocked_get_all.return_value = self.mocked_responses
-        data_list = ES.fetch_exchange(
-            ZoneKey("ES-IB-FO"), ZoneKey("ES-IB-IZ"), self.session
+        # Test "get" function has been called correctly
+        self.assertEqual(
+            mocked_session_get.call_args[0][0],
+            "https://demanda.ree.es/WSvisionaMovilesBalearesRest/resources/demandaGeneracionBaleares?curva=MENORCA5M&fecha=2023-09-03",
+        )
+
+        # Test that the data is parsed correctly afterwards
+        self.assertEqual(len(data_list), 1)
+        self.assertEqual(data_list[0]["zoneKey"], "ES-IB-ME")
+        self.assertEqual(data_list[0]["source"], "demanda.ree.es")
+        self.assertEqual(
+            data_list[0]["production"],
+            {
+                "biomass": 0.0,
+                "coal": 0.0,
+                "gas": 0.0,
+                "oil": 81.9,
+                "solar": 1.1,
+                "unknown": 0.0,
+                "wind": 0.2,
+            },
         )
-        self.assertEqual(len(data_list), 2)
-        for data in data_list:
-            self.assertEqual(data["sortedZoneKeys"], "ES-IB-FO->ES-IB-IZ")
-            self.assertEqual(data["source"], "demanda.ree.es")
-            self.assertIsNotNone(data["netFlow"])
-            self.assertEqual(data["netFlow"], -10.0)
-            self.assertTrue(isinstance(data["datetime"], datetime))
+        self.assertTrue(isinstance(data_list[0]["datetime"], datetime))
         self.assertEqual(
             data_list[0]["datetime"],
-            datetime.utcfromtimestamp(1687192328).replace(tzinfo=timezone.utc),
+            datetime(2023, 9, 2, 19, 0, tzinfo=timezone.utc),
         )
 
     ### Mallorca
     # Exchange
-    @patch.object(Mallorca, "get_all")
-    def test_fetch_exchange(self, mocked_get_all):
-        mocked_get_all.return_value = self.mocked_responses
-        data_list = ES.fetch_exchange(ZoneKey("ES"), ZoneKey("ES-IB-MA"), self.session)
-        self.assertEqual(len(data_list), 2)
-        for data in data_list:
-            self.assertEqual(data["sortedZoneKeys"], "ES->ES-IB-MA")
-            self.assertEqual(data["source"], "demanda.ree.es")
-            self.assertIsNotNone(data["netFlow"])
-            self.assertEqual(data["netFlow"], 50.0)
-            self.assertTrue(isinstance(data["datetime"], datetime))
+    @patch("requests.Response")
+    @patch("parsers.ES.Session.get")
+    def test_fetch_exchange(self, mocked_session_get, mocked_response):
+        mocked_response.ok = True
+        mocked_response.text = r'null({"valoresHorariosGeneracion":[{"ts":"2023-09-03 19:05","dem":670.3,"car":0.0,"die":0.0,"gas":0.0,"cc":416.4,"cb":309.8,"fot":19.7,"tnr":0.0,"trn":0.0,"eol":0.0,"emm":-20.6,"emi":-91.3,"otrRen":0.0,"resid":34.6,"genAux":0.0,"cogen":1.7,"eif":-0.0}]}'
+        mocked_session_get.return_value = mocked_response
+        data_list = ES.fetch_exchange(
+            ZoneKey("ES"),
+            ZoneKey("ES-IB-MA"),
+            self.session,
+            datetime.fromisoformat("2023-09-03"),
+        )
+
+        # Test "get" function has been called correctly
+        self.assertEqual(
+            mocked_session_get.call_args[0][0],
+            "https://demanda.ree.es/WSvisionaMovilesBalearesRest/resources/demandaGeneracionBaleares?curva=MALLORCA5M&fecha=2023-09-03",
+        )
+
+        # Test that the data is parsed correctly afterwards
+        self.assertEqual(len(data_list), 1)
+        self.assertEqual(data_list[0]["sortedZoneKeys"], "ES->ES-IB-MA")
+        self.assertEqual(data_list[0]["source"], "demanda.ree.es")
+        self.assertIsNotNone(data_list[0]["netFlow"])
+        self.assertEqual(data_list[0]["netFlow"], 309.8)
+        self.assertTrue(isinstance(data_list[0]["datetime"], datetime))
         self.assertEqual(
             data_list[0]["datetime"],
-            datetime.utcfromtimestamp(1687192328).replace(tzinfo=timezone.utc),
+            datetime(2023, 9, 3, 17, 5, tzinfo=timezone.utc),
         )