Repository: electricitymaps/electricitymaps-contrib

Issue:
MN parser down
## Description
Mongolia parser is down, but the bot hasn't open an issue. I am opening this one to alert the maintainers.
It seems this time the parser is down because of a change in the json that was being parsed:
> raise ParserException(
parsers.lib.exceptions.ParserException: MN.py Parser: Fetched keys from source dict_keys(['date', 'syssum', 'tpp', 'sumnar', 'sums', 'energyimport', 't']) do not match expected keys dict_values(['date', 'syssum', 'sumnar', 'sums', 'energyimport', 't']).

A new key called tpp (thermal power plants?) has being added. The value of this new key doesn't match the previously calculated unknown production (so tpp plus other keys don't add up to consumption). What should be done to fix this? It seems an unknown source is being added.

By the way, a bit off-topic, but I have noticed that the Mongolia parser outputs global exchange data. We currently get the exchange with Russia from its parser, so we could calculate the exchange with China by substracting the other exchange. Is this possible?


Discussion:
The moldova parser (MD.py) considers "tpp" as gas. Not sure if the same assumption should apply here though. Wikipedia says Mongolia's electricity was 98% from coal (in 2012 for what it's worth), which would point to this "tpp" number representing coal: https://en.wikipedia.org/wiki/Energy_in_Mongolia 

If it is coal, I have a proposed fix ready. 

Maybe someone more familiar with these parsers / data could weigh in?
> The moldova parser (MD.py) considers "tpp" as gas. Not sure if the same assumption should apply here though. Wikipedia says Mongolia's electricity was 98% from coal (in 2012 for what it's worth), which would point to this "tpp" number representing coal: https://en.wikipedia.org/wiki/Energy_in_Mongolia
> 
> If it is coal, I have a proposed fix ready.
> 
> Maybe someone more familiar with these parsers / data could weigh in?

Looks like it is a mix of coal and oil if we look at IEA data from 2020 which means we can't assign it all to coal. But what we can do is assign it to unknown and try and use it to get the hydro share instead.

Since the current data contains wind, solar, thermal, imports and the consumption and Mongolia only have production from solar, wind, hydro, coal and oil we should be able to use that to get the hydro share with some simple math. It's not ideal but it's better than nothing.

(We have to verify it all adds up though).
> By the way, a bit off-topic, but I have noticed that the Mongolia parser outputs global exchange data. We currently get the exchange with Russia from its parser, so we could calculate the exchange with China by substracting the other exchange. Is this possible?

If the datetimes for the data match between the two sources it can probably be done but otherwise it is a lot trickier.
> > By the way, a bit off-topic, but I have noticed that the Mongolia parser outputs global exchange data. We currently get the exchange with Russia from its parser, so we could calculate the exchange with China by substracting the other exchange. Is this possible?
> 
> If the datetimes for the data match between the two sources it can probably be done but otherwise it is a lot trickier.

They do! Checking so I've discovered that currently the datetime is saved incorrectly (because of the use of .replace with a pytz timezone, i solved it by using a ZoneInfo object)
This causes the timezone to be +07:08 (yes, 8 minutes!)

> > The moldova parser (MD.py) considers "tpp" as gas. Not sure if the same assumption should apply here though. Wikipedia says Mongolia's electricity was 98% from coal (in 2012 for what it's worth), which would point to this "tpp" number representing coal: https://en.wikipedia.org/wiki/Energy_in_Mongolia
> > If it is coal, I have a proposed fix ready.
> > Maybe someone more familiar with these parsers / data could weigh in?
> 
> Looks like it is a mix of coal and oil if we look at IEA data from 2020 which means we can't assign it all to coal. But what we can do is assign it to unknown and try and use it to get the hydro share instead.
> 
> Since the current data contains wind, solar, thermal, imports and the consumption and Mongolia only have production from solar, wind, hydro, coal and oil we should be able to use that to get the hydro share with some simple math. It's not ideal but it's better than nothing.
> 
> (We have to verify it all adds up though).

I am sorry to say that it doesn't add up:

This is the generation capacity according to [https://ndc.energy.mn/](https://ndc.energy.mn/):

![Screenshot_20230921_005014](https://github.com/electricitymaps/electricitymaps-contrib/assets/84802276/2bc2edc2-6259-4f7b-b681-c4ca80b54369)

However, many times the production left after substracting the other sources is over 23 MW. It is possible "tpp" only takes into account the power plants listed, meaning the rest comes from small coal, waste, oil or hydro power plants. We can be sure there are more plants than listed because the ones listed add up to 1532 MW of capacity.

I think we should fix the parser to work as it was doing previously, ignoring the tpp value until it is clear what all the keys mean exactly. Also to note is that the page contains several other categories, which may become populated in the future. Hydro is one of these categories:

![Screenshot_20230921_011817](https://github.com/electricitymaps/electricitymaps-contrib/assets/84802276/a55719af-baa1-45d2-b08e-67661c30f844)

(In case it is not clear, the image is powered by the data from the source the parser uses.)

Resolving patch:
diff --git a/parsers/MN.py b/parsers/MN.py
--- a/parsers/MN.py
+++ b/parsers/MN.py
@@ -29,18 +29,18 @@
 }
 
 
-def parse_json(web_json: dict) -> dict[str, Any]:
+def parse_json(web_json: dict, logger: Logger, zone_key: ZoneKey) -> dict[str, Any]:
     """
     Parse the fetched JSON data to our query format according to JSON_QUERY_TO_SRC.
     Example of expected JSON format present at URL:
-    {"date":"2023-06-27 18:00:00","syssum":"869.37","sumnar":42.34,"sums":119.79,"energyimport":"49.58","t":"17"}
+    {"date":"2023-06-27 18:00:00","syssum":"869.37","sumnar":42.34,"sums":119.79,"energyimport":"49.58"}
     """
 
     # Validate first if keys in fetched dict match expected keys
     if set(JSON_QUERY_TO_SRC.values()) != set(web_json.keys()):
-        raise ParserException(
-            parser="MN.py",
-            message=f"Fetched keys from source {web_json.keys()} do not match expected keys {JSON_QUERY_TO_SRC.values()}.",
+        logger.error(
+            msg=f"Fetched keys from source {web_json.keys()} do not match expected keys {JSON_QUERY_TO_SRC.values()}.",
+            extra={"zone_key": zone_key, "parser": "MN.py"},
         )
 
     if None in web_json.values():
@@ -52,7 +52,7 @@ def parse_json(web_json: dict) -> dict[str, Any]:
     # Then we can safely parse them
     query_data = dict()
     for query_key, src_key in JSON_QUERY_TO_SRC.items():
-        if query_key == "time":
+        if "time" in query_key:
             # convert to datetime
             query_data[query_key] = datetime.fromisoformat(web_json[src_key]).replace(
                 tzinfo=TZ
@@ -64,7 +64,7 @@ def parse_json(web_json: dict) -> dict[str, Any]:
     return query_data
 
 
-def query(session: Session) -> dict[str, Any]:
+def query(session: Session, logger: Logger, zone_key: ZoneKey) -> dict[str, Any]:
     """
     Query the JSON endpoint and parse it.
     """
@@ -79,7 +79,7 @@ def query(session: Session) -> dict[str, Any]:
 
     # Read as JSON
     response_json = target_response.json()
-    query_result = parse_json(response_json)
+    query_result = parse_json(response_json, logger, zone_key)
 
     return query_result
 
@@ -93,11 +93,11 @@ def fetch_production(
     if target_datetime:
         raise NotImplementedError("This parser is not yet able to parse past dates.")
 
-    query_data = query(session)
+    query_data = query(session, logger, zone_key)
 
-    # Calculated 'unknown' production from available data (consumption, import, solar, wind).
+    # Calculated 'unknown' production from available data (consumption, import, solar, wind, tpp).
     # 'unknown' consists of 92.8% coal, 5.8% oil and 1.4% hydro as per 2020; sources: IEA and IRENA statistics.
-    query_data["unknownMW"] = round(
+    query_data["leftoverMW"] = round(
         query_data["consumptionMW"]
         - query_data["importMW"]
         - query_data["solarMW"]
@@ -105,11 +105,10 @@ def fetch_production(
         13,
     )
 
-    prod_mix = ProductionMix(
-        solar=query_data["solarMW"],
-        wind=query_data["windMW"],
-        unknown=query_data["unknownMW"],
-    )
+    prod_mix = ProductionMix()
+    prod_mix.add_value("solar", query_data["solarMW"])
+    prod_mix.add_value("wind", query_data["windMW"])
+    prod_mix.add_value("unknown", query_data["leftoverMW"])
 
     prod_breakdown_list = ProductionBreakdownList(logger)
     prod_breakdown_list.append(
@@ -131,7 +130,7 @@ def fetch_consumption(
     if target_datetime:
         raise NotImplementedError("This parser is not yet able to parse past dates.")
 
-    query_data = query(session)
+    query_data = query(session, logger, zone_key)
 
     consumption_list = TotalConsumptionList(logger)
     consumption_list.append(


Test patch: