Repository: electricitymaps/electricitymaps-contrib

Issue:
Switch Ontario to using IESO XML data
Use http://reports.ieso.ca/public/IntertieScheduleFlow/PUB_IntertieScheduleFlow_20180108.xml (note that URL changes, using Toronto timezone = UTC-5 / -4) rather than http://live.gridwatch.ca/WebServices/GridWatchWebApp.asmx/GetHomeViewData_v2

Advantages:
* breakdown for every 5 minutes (in browser view, hover/tap on the "flow" values for each hour to see breakdown)
* the "flow" values appear to take into account Lake Erie Loop between Michigan and New York - basically "flow" is the actual electrons on the wire, "exp" is the accounting value

First pointed out by @HansHyde in https://github.com/tmrowco/electricitymap/pull/925#issuecomment-353621704

edit: also use IESO data for generation mix, as it is more official. Live data is at http://www.ieso.ca/en/power-data?chart=supply and historical as XMLO at http://reports.ieso.ca/public/GenOutputCapability/


Discussion:
@jarek 

The NY - Ontario exchange is also available in 5min increments here... http://mis.nyiso.com/public/csv/ExternalLimitsFlows/currentExternalLimitsFlows.csv

represented by this line here... "01/08/2018 08:10,SCH - OH - NY,23317,904.9,1750,-1900"

The "OH" must represent "Ontario Hydro" its name prior to deregulation and the IESO being formed, as opposed to "OH" equal Ohio
Any status here?
I was looking at this, but haven't had time yet. It should still be done, if only to get the MISO/NYISO Lake Erie Loop-affected exchanges correct.

I think IESO also have historical generation mix available through XML files that we can switch to, as well. Their generation data is at http://www.ieso.ca/en/power-data?chart=supply and historical at http://reports.ieso.ca/public/GenOutputCapability/
For whatever reason, the Quebec-Ontario interchange is wrongly showing as 0 in Electricity Map. http://live.gridwatch.ca reports Ontario importing total ~1.1 GW from Quebec at 14:00-15:00 2018-07-05, and http://reports.ieso.ca/public/IntertieScheduleFlow/PUB_IntertieScheduleFlow_20180705.xml says ~1.4 GW, but we have 0. For that matter it's been 0 whenever I looked at EM in the past few weeks.
In looking at this, I found the cause of the Quebec bug: https://github.com/tmrowco/electricitymap-contrib/blob/6b3f2dd8669db10a05b3a92bc0441ef778014d2f/parsers/CA_ON.py#L179 makes `keys` an iterator, but we then attempt to consume it twice: to get the datetime and then the netFlow. The problem is by the time we get to the netFlow, the iterator is completely consumed and will be empty. `map` will happily run over a consumed iterator, simply returning an empty sequence.

So it looks like import from Quebec would have been 0 from day 1 (since https://github.com/tmrowco/electricitymap-contrib/commit/202383b244d2b6047148c3289f01dac6f419c971)

I will try to reimplement using IESO data. It looks like 3 months' history is reachable (currently http://reports.ieso.ca/public/IntertieScheduleFlow/PUB_IntertieScheduleFlow_20180414.xml loads, 20180413 doesn't), perhaps it can be recrawled.
@jarek can we close this now?
I am working on changing the production parser so that it can handle target_datetime (for up to 3 months in the past). We can close this ticket if you like, I have the URL saved now :)
If anyone else is wondering: this "choppy" import chart today is because 0 MW values on the full hour were inserted by the old code. After deploy of #1515, the corrected values since midnight Toronto time were fetched with every-5-minute timestamps, but the older 0 values were not overwritten, resulting in 0 import from QC on full hour, and correct values in between.

The same is also the case with exports to MISO and NY (the full-hour values are the accounting values), though not visible on a chart, only by scrolling through history.

This will not be an issue going forward as only the new code will be used.

If you wish to re-crawl the past 3 months, you should first invalidate the values for CA-ON->CA-QC currently in the database for before today.

![image](https://user-images.githubusercontent.com/47415/43341089-b63a5988-91de-11e8-9a71-44c68cd41c77.png)


Resolving patch:
diff --git a/parsers/CA_ON.py b/parsers/CA_ON.py
--- a/parsers/CA_ON.py
+++ b/parsers/CA_ON.py
@@ -1,19 +1,44 @@
 #!/usr/bin/env python3
 
-from collections import defaultdict
-import datetime
+from datetime import timedelta, timezone
+import logging
+import xml.etree.ElementTree as ET
 
 # The arrow library is used to handle datetimes
 import arrow
 
-# pytz gets tzinfo objects
-import pytz
-
 # The request library is used to fetch content through HTTP
 import requests
 
-from bs4 import BeautifulSoup
+# pandas processes tabular data
+import pandas as pd
+
+"""
+Some notes about timestamps:
+
+IESO website says:
+"The IESO uses the "Hour Ending" naming convention for the hours in a day. For example, Hour 1 is from 12 am. to 1 am.,
+Hour 2 is from 1 am. to 2 am. Hours 1-24 are the hours from midnight one day through midnight the next day."
+
+Observations:
+- At 13:53, HOEP report will have data with "Hour" column from 1 to 13.
+  Output and Capability report will have data with "Hour" column from 1 to 13.
+  Intertie Flow report will have data with "Hour" column from 1 to 13 *and* Interval column
+  from 1 to 12 for all of these hours, including the 13th hour.
+- At 14:18, HOEP report will go up to 14, Output report will go up to 14,
+  but update for Intertie report is not yet updated.
+- At 14:31, Intertie report is updated with Hour 14 which has Intervals 1 to 12.
+
+In the script, the Intertie report is shifted 1 hour and 5 minutes back, so that it lines up with
+the production and price data availability.
+"""
+
+# IESO says "Eastern Standard Time is used year round."
+# This would mean daylight savings is not used (that is called "Eastern Daylight Time"),
+# and we need to use UTC-5 rather than 'Canada/Eastern'
+tz_obj = timezone(timedelta(hours=-5), name='UTC-5')
 
+# fuel types used by IESO
 MAP_GENERATION = {
     'BIOFUEL': 'biomass',
     'GAS': 'gas',
@@ -23,21 +48,64 @@
     'WIND': 'wind'
 }
 
-timezone = 'Canada/Eastern'
-tz_obj = pytz.timezone(timezone)
+# exchanges and sub-exchanges used by IESO
+MAP_EXCHANGE = {
+    'MANITOBA': 'CA-MB->CA-ON',
+    'MANITOBA SK': 'CA-MB->CA-ON',
+    'MICHIGAN': 'CA-ON->US-MISO',
+    'MINNESOTA': 'CA-ON->US-MISO',
+    'NEW-YORK': 'CA-ON->US-NY',
+    'PQ.AT': 'CA-ON->CA-QC',
+    'PQ.B5D.B31L': 'CA-ON->CA-QC',
+    'PQ.D4Z': 'CA-ON->CA-QC',
+    'PQ.D5A': 'CA-ON->CA-QC',
+    'PQ.H4Z': 'CA-ON->CA-QC',
+    'PQ.H9A': 'CA-ON->CA-QC',
+    'PQ.P33C': 'CA-ON->CA-QC',
+    'PQ.Q4C': 'CA-ON->CA-QC',
+    'PQ.X2Y': 'CA-ON->CA-QC'
+}
+
+PRODUCTION_URL = 'http://reports.ieso.ca/public/GenOutputCapability/PUB_GenOutputCapability_{YYYYMMDD}.xml'
+PRICE_URL = 'http://reports.ieso.ca/public/DispUnconsHOEP/PUB_DispUnconsHOEP_{YYYYMMDD}.xml'
+EXCHANGES_URL = 'http://reports.ieso.ca/public/IntertieScheduleFlow/PUB_IntertieScheduleFlow_{YYYYMMDD}.xml'
+
+XML_NS_TEXT = '{http://www.theIMO.com/schema}'
+
+
+def _fetch_ieso_xml(target_datetime, session, logger, url_template):
+    dt = arrow.get(target_datetime).to(tz_obj).replace(hour=0, minute=0, second=0, microsecond=0)
+
+    r = session or requests.session()
+    url = url_template.format(YYYYMMDD=dt.format('YYYYMMDD'))
+    response = r.get(url)
+
+    if not response.ok:
+        # Data is generally available for past 3 months. Requesting files older than this
+        # returns an HTTP 404 error.
+        logger.info('CA-ON: failed getting requested data for datetime {} from IESO server - URL {}'.format(dt, url))
+        return dt, None
 
+    xml = ET.fromstring(response.text)
 
-def fetch_production(zone_key='CA-ON', session=None, target_datetime=None, logger=None):
-    """Requests the last known production mix (in MW) of a given country
+    return dt, xml
+
+
+def fetch_production(zone_key='CA-ON', session=None, target_datetime=None,
+                     logger=logging.getLogger(__name__)):
+    """Requests the last known production mix (in MW) of a given region
 
     Arguments:
-    zone_key (optional) -- used in case a parser is able to fetch multiple countries
-    session (optional)      -- request session passed in order to re-use an existing session
+    zone_key (optional): ignored here, only information for CA-ON is returned
+    session (optional): request session passed in order to re-use an existing session
+    target_datetime: the datetime for which we want production data. If not provided, we should
+      default it to now. The provided target_datetime is timezone-aware in UTC.
+    logger: an instance of a `logging.Logger`; all raised exceptions are also logged automatically
 
     Return:
-    A dictionary in the form:
+    A list of dictionaries in the form:
     {
-      'zoneKey': 'FR',
+      'zoneKey': 'CA-ON',
       'datetime': '2017-01-01T00:00:00Z',
       'production': {
           'biomass': 0.0,
@@ -57,47 +125,78 @@ def fetch_production(zone_key='CA-ON', session=None, target_datetime=None, logge
       'source': 'mysource.com'
     }
     """
-    if target_datetime:
-        raise NotImplementedError('This parser is not yet able to parse past dates')
-    
-    r = session or requests.session()
-    url = 'http://www.ieso.ca/-/media/files/ieso/uploaded/chart/generation_fuel_type_multiday.xml?la=en'
-    response = r.get(url)
-    soup = BeautifulSoup(response.text, 'html.parser')
-
-    data = {}
-
-    start_datetime = arrow.get(
-        arrow.get(soup.find_all('startdate')[0].contents[0]).datetime, timezone)
-
-    # Iterate over all datasets (production types)
-    for item in soup.find_all('dataset'):
-        key = item.attrs['series']
-        for rowIndex, row in enumerate(item.find_all('value')):
-            if not len(row.contents):
-                continue
-            if rowIndex not in data:
-                data[rowIndex] = {
-                    'datetime': start_datetime.replace(hours=+rowIndex).datetime,
-                    'zoneKey': zone_key,
-                    'production': {
-                        'coal': 0
-                    },
-                    'storage': {},
-                    'source': 'ieso.ca',
-                }
-            data[rowIndex]['production'][MAP_GENERATION[key]] = \
-                float(row.contents[0])
-
-    return [data[k] for k in sorted(data.keys())]
-
-
-def fetch_price(zone_key='CA-ON', session=None, target_datetime=None, logger=None):
-    """Requests the last known power price of a given country
+
+    dt, xml = _fetch_ieso_xml(target_datetime, session, logger, PRODUCTION_URL)
+
+    if not xml:
+        return []
+
+    generators = xml\
+        .find(XML_NS_TEXT + 'IMODocBody')\
+        .find(XML_NS_TEXT + 'Generators')\
+        .findall(XML_NS_TEXT + 'Generator')
+
+    def production_or_zero(output):
+        # Sometimes the XML data has no "EnergyMW" tag for a given plant at a given hour.
+        # The report XSL formats this as "N/A" - we return 0
+        tag = output.find(XML_NS_TEXT + 'EnergyMW')
+        if tag is not None:
+            return tag.text
+        else:
+            return 0
+
+    # flat iterable of per-generator-plant productions per time from the XML data
+    all_productions = (
+        {
+            'name': generator.find(XML_NS_TEXT + 'GeneratorName').text,
+            'fuel': MAP_GENERATION[
+                generator.find(XML_NS_TEXT + 'FuelType').text
+            ],
+            'dt': dt.replace(hours=+int(
+                output.find(XML_NS_TEXT + 'Hour').text
+            )).datetime,
+            'production': float(production_or_zero(output))
+        }
+        for generator in generators
+        for output in generator.find(XML_NS_TEXT + 'Outputs').findall(XML_NS_TEXT + 'Output')
+    )
+
+    df = pd.DataFrame(all_productions)
+
+    # group individual plants using the same fuel together for each time period
+    by_fuel = df.groupby(['dt', 'fuel']).sum().unstack()
+    # to debug, you can `print(by_fuel)` here, which gives a very pretty table
+
+    by_fuel_dict = by_fuel['production'].to_dict('index')
+
+    data = [
+        {
+            'datetime': time.to_pydatetime(),
+            'zoneKey': zone_key,
+            'production': productions,
+            'storage': {},
+            'source': 'ieso.ca',
+        }
+        for time, productions in by_fuel_dict.items()
+    ]
+
+    # being constructed from a dict, data is not guaranteed to be in chronological order.
+    # sort it for clean-ness and easier debugging.
+    data = sorted(data, key=lambda dp: dp['datetime'])
+
+    return data
+
+
+def fetch_price(zone_key='CA-ON', session=None, target_datetime=None,
+                logger=logging.getLogger(__name__)):
+    """Requests the last known power price per MWh of a given region
 
     Arguments:
-    zone_key (optional) -- used in case a parser is able to fetch multiple countries
-    session (optional)      -- request session passed in order to re-use an existing session
+    zone_key: ignored here, only information for CA-ON is returned
+    session: requests session passed in order to re-use an existing session,
+    target_datetime: the datetime for which we want production data. If not provided, we should
+      default it to now. The provided target_datetime is timezone-aware in UTC.
+    logger: an instance of a `logging.Logger`; all raised exceptions are also logged automatically
 
     Return:
     A dictionary in the form:
@@ -109,47 +208,44 @@ def fetch_price(zone_key='CA-ON', session=None, target_datetime=None, logger=Non
       'source': 'mysource.com'
     }
     """
-    if target_datetime:
-        raise NotImplementedError('This parser is not yet able to parse past dates')
 
-    r = session or requests.session()
-    url = 'http://www.ieso.ca/-/media/files/ieso/uploaded/chart/price_multiday.xml?la=en'
-    response = r.get(url)
-    soup = BeautifulSoup(response.text, 'html.parser')
-
-    data = {}
-
-    start_datetime = arrow.get(
-        arrow.get(soup.find_all('startdate')[0].contents[0]).datetime, timezone)
-
-    # Iterate over all datasets (production types)
-    for item in soup.find_all('dataset'):
-        key = item.attrs['series']
-        if key != 'HOEP':
-            continue
-        for rowIndex, row in enumerate(item.find_all('value')):
-            if not len(row.contents):
-                continue
-            if rowIndex not in data:
-                data[rowIndex] = {
-                    'datetime': start_datetime.replace(hours=+rowIndex).datetime,
-                    'zoneKey': zone_key,
-                    'currency': 'CAD',
-                    'source': 'ieso.ca',
-                }
-            data[rowIndex]['price'] = \
-                float(row.contents[0])
-
-    return [data[k] for k in sorted(data.keys())]
+    # "HOEP" below is "Hourly Ontario Energy Price".
+    # There also exists a 5-minute price, but its archives only go back roughly 4 days
+    # (see http://www.ieso.ca/power-data "5 Minute Market Clearing Price").
+
+    dt, xml = _fetch_ieso_xml(target_datetime, session, logger, PRICE_URL)
+
+    if not xml:
+        return []
+
+    prices = xml\
+        .find(XML_NS_TEXT + 'IMODocBody')\
+        .find(XML_NS_TEXT + 'HOEPs')\
+        .findall(XML_NS_TEXT + 'HOEP')
+
+    data = [
+        {
+            'datetime': dt.replace(hours=+int(
+                price.find(XML_NS_TEXT + 'Hour').text
+            )).datetime,
+            'price': float(price.find(XML_NS_TEXT + 'Price').text),
+            'currency': 'CAD',
+            'source': 'ieso.ca',
+            'zoneKey': zone_key
+        }
+        for price in prices
+    ]
 
     return data
 
 
-def fetch_exchange(zone_key1, zone_key2, session=None, target_datetime=None, logger=None):
-    """Requests the last known power exchange (in MW) between two countries
+def fetch_exchange(zone_key1, zone_key2, session=None, target_datetime=None,
+                   logger=logging.getLogger(__name__)):
+    """Requests the last known power exchange (in MW) between two regions
 
     Arguments:
-    zone_key: used in case a parser is able to fetch multiple zones
+    zone_key1: the first region code
+    zone_key2: the second region code; order of the two codes in params doesn't matter
     session: requests session passed in order to re-use an existing session,
     target_datetime: the datetime for which we want production data. If not provided, we should
       default it to now. The provided target_datetime is timezone-aware in UTC.
@@ -165,80 +261,72 @@ def fetch_exchange(zone_key1, zone_key2, session=None, target_datetime=None, log
     }]
     """
 
-    exchange_maps = {
-        'MANITOBA': 'CA-MB->CA-ON',
-        'MANITOBA SK': 'CA-MB->CA-ON',
-        'MICHIGAN': 'CA-ON->US-MISO',
-        'MINNESOTA': 'CA-ON->US-MISO',
-        'NEW-YORK': 'CA-ON->US-NY',
-        'PQ.AT': 'CA-ON->CA-QC',
-        'PQ.B5D.B31L': 'CA-ON->CA-QC',
-        'PQ.D4Z': 'CA-ON->CA-QC',
-        'PQ.D5A': 'CA-ON->CA-QC',
-        'PQ.H4Z': 'CA-ON->CA-QC',
-        'PQ.H9A': 'CA-ON->CA-QC',
-        'PQ.P33C': 'CA-ON->CA-QC',
-        'PQ.Q4C': 'CA-ON->CA-QC',
-        'PQ.X2Y': 'CA-ON->CA-QC'
-    }
-
     sorted_zone_keys = '->'.join(sorted([zone_key1, zone_key2]))
 
-    if sorted_zone_keys not in exchange_maps.values():
+    if sorted_zone_keys not in MAP_EXCHANGE.values():
         raise NotImplementedError('This exchange pair is not implemented')
 
-    dt = arrow.get(target_datetime).to(tz_obj)
-    filename = dt.format('YYYYMMDD')
+    dt, xml = _fetch_ieso_xml(target_datetime, session, logger, EXCHANGES_URL)
 
-    r = session or requests.session()
-    url = 'http://reports.ieso.ca/public/IntertieScheduleFlow/PUB_IntertieScheduleFlow_{}.xml'.format(filename)
-    response = r.get(url)
-    soup = BeautifulSoup(response.text, 'html.parser')
-
-    interties = soup.find_all('intertiezone')
+    if not xml:
+        return []
 
-    sought_intertie_flows = defaultdict(list)
+    intertie_zones = xml\
+        .find(XML_NS_TEXT + 'IMODocBody')\
+        .findall(XML_NS_TEXT + 'IntertieZone')
 
-    for intertie in interties:
-        intertie_name = intertie.find('intertiezonename').text
+    def flow_dt(flow):
+        # The IESO day starts with Hour 1, Interval 1.
+        # At 11:37, we might have data for up to Hour 11, Interval 12.
+        # This means it is necessary to subtract 1 from the values
+        # (otherwise we would have data for 12:00 already at 11:37).
+        hour = int(flow.find(XML_NS_TEXT + 'Hour').text) - 1
+        minute = (int(flow.find(XML_NS_TEXT + 'Interval').text) - 1) * 5
 
-        if intertie_name not in exchange_maps:
-            logger.warning('CA-ON: unrecognized intertie name {}, please implement it!'.format(intertie_name))
-            continue
+        return dt.replace(hours=+hour, minutes=+minute).datetime
 
-        mapping = exchange_maps[intertie_name]
+    # flat iterable of per-intertie values per time from the XML data
+    all_exchanges = (
+        {
+            'name': intertie.find(XML_NS_TEXT + 'IntertieZoneName').text,
+            'dt': flow_dt(flow),
+            'flow': float(flow.find(XML_NS_TEXT + 'Flow').text)
+        }
+        for intertie in intertie_zones
+        for flow in intertie.find(XML_NS_TEXT + 'Actuals').findall(XML_NS_TEXT + 'Actual')
+    )
 
-        if not mapping == sorted_zone_keys:
-            # we're not interested in data for this zone, skip it
-            continue
+    df = pd.DataFrame(all_exchanges)
 
-        # in the XML, flow into Ontario is always negative.
-        # in EM, for 'CA-MB->CA-ON', flow into Ontario is positive.
-        if mapping.startswith('CA-ON->'):
-            direction = 1
-        else:
-            direction = -1
+    # verify that there haven't been new exchanges or sub-exchanges added
+    all_exchange_names = set(df['name'].unique())
+    known_exchange_names = set(MAP_EXCHANGE.keys())
+    unknown_exchange_names = all_exchange_names - known_exchange_names
+    if unknown_exchange_names:
+        logger.warning('CA-ON: unrecognized intertie name(s) {}, please implement!'.format(unknown_exchange_names))
 
-        actuals = intertie.find_all('actual')
+    # filter to only the sought exchanges
+    sought_exchanges = [ieso_name for ieso_name, em_name in MAP_EXCHANGE.items() if sorted_zone_keys == em_name]
+    df = df[df['name'].isin(sought_exchanges)]
 
-        for actual in actuals:
-            hour = int(actual.find('hour').text) - 1
-            minute = (int(actual.find('interval').text) - 1) * 5
-            flow = float(actual.find('flow').text) * direction
+    # in the XML, flow into Ontario is always negative.
+    # in EM, for 'CA-MB->CA-ON', flow into Ontario is positive.
+    if not sorted_zone_keys.startswith('CA-ON->'):
+        df['flow'] *= -1
 
-            dt_aware = datetime.datetime(dt.year, dt.month, dt.day, hour, minute, tzinfo=tz_obj)
+    # group flows for sub-interchanges together and sum them
+    grouped = df.groupby(['dt']).sum()
 
-            sought_intertie_flows[dt_aware].append(flow)
+    grouped_dict = grouped['flow'].to_dict()
 
-    # add up values for same datetime for exchanges with more than one intertie
     data = [
         {
-            'datetime': flow_dt,
+            'datetime': flow_dt.to_pydatetime(),
             'sortedZoneKeys': sorted_zone_keys,
-            'netFlow': sum(flow_figures),
-            'source': 'ieso.ca'
+            'netFlow': flow,
+            'source': 'ieso.ca',
         }
-        for flow_dt, flow_figures in sought_intertie_flows.items()
+        for flow_dt, flow in grouped_dict.items()
     ]
 
     # being constructed from a dict, data is not guaranteed to be in chronological order.
@@ -251,18 +339,50 @@ def fetch_exchange(zone_key1, zone_key2, session=None, target_datetime=None, log
 if __name__ == '__main__':
     """Main method, never used by the Electricity Map backend, but handy for testing."""
 
+    now = arrow.utcnow()
+
     print('fetch_production() ->')
     print(fetch_production())
+
+    print('we expect correct results when time in UTC and Ontario differs')
+    print('data should be for {}'.format(now.replace(hour=2).to(tz_obj).format('YYYY-MM-DD')))
+    print('fetch_production("CA-ON", target_datetime=now.replace(hour=2)) ->')
+    print(fetch_production("CA-ON", target_datetime=now.replace(hour=2)))
+
+    print('we expect results for 2 months ago')
+    print('fetch_production(target_datetime=now.shift(months=-2).datetime)) ->')
+    print(fetch_production(target_datetime=now.shift(months=-2).datetime))
+
+    print('there are likely no results for 2 years ago')
+    print('fetch_production(target_datetime=now.shift(years=-2).datetime)) ->')
+    print(fetch_production(target_datetime=now.shift(years=-2).datetime))
+
     print('fetch_price() ->')
     print(fetch_price())
+
+    print('we expect correct results when time in UTC and Ontario differs')
+    print('data should be for {}'.format(now.replace(hour=2).to(tz_obj).format('YYYY-MM-DD')))
+    print('fetch_price(target_datetime=now.replace(hour=2)) ->')
+    print(fetch_price(target_datetime=now.replace(hour=2)))
+
+    print('we expect results for 2 months ago')
+    print('fetch_price(target_datetime=now.shift(months=-2).datetime)) ->')
+    print(fetch_price(target_datetime=now.shift(months=-2).datetime))
+
+    print('there are likely no results for 2 years ago')
+    print('fetch_price(target_datetime=now.shift(years=-2).datetime)) ->')
+    print(fetch_price(target_datetime=now.shift(years=-2).datetime))
+
     print('fetch_exchange("CA-ON", "US-NY") ->')
     print(fetch_exchange("CA-ON", "US-NY"))
 
-    now = arrow.utcnow()
-
     print('fetch_exchange("CA-ON", "CA-QC", target_datetime=now.datetime)) ->')
     print(fetch_exchange("CA-ON", "CA-QC", target_datetime=now.datetime))
 
+    print('test Ontario-to-Manitoba, this must be opposite sign from reported IESO values')
+    print('fetch_exchange("CA-ON", "CA-MB", target_datetime=now.datetime)) ->')
+    print(fetch_exchange("CA-ON", "CA-MB", target_datetime=now.datetime))
+
     print('we expect correct results when time in UTC and Ontario differs')
     print('data should be for {}'.format(now.replace(hour=2).to(tz_obj).format('YYYY-MM-DD')))
     print('fetch_exchange("CA-ON", "CA-QC", target_datetime=now.replace(hour=2)) ->')
@@ -275,3 +395,7 @@ def fetch_exchange(zone_key1, zone_key2, session=None, target_datetime=None, log
     print('there are likely no results for 2 years ago')
     print('fetch_exchange("CA-ON", "CA-QC", target_datetime=now.shift(years=-2).datetime)) ->')
     print(fetch_exchange("CA-ON", "CA-QC", target_datetime=now.shift(years=-2).datetime))
+
+    print('requesting an exchange with Nova Scotia should raise exception')
+    print('fetch_exchange("CA-ON", "CA-NS")) ->')
+    print(fetch_exchange("CA-ON", "CA-NS"))


Test patch: