Repository: IAMconsortium/pyam
Pull request #285 (2019-11-07 14:07:08)

Add ability to request regions with synonyms from IXMP web API
Changes
- (optionally) fetch region synonyms
- fix fetching timeseries data explicitly specifying column types

---
- [x] Tests Added
- [x] Documentation Added
- [x] Description in RELEASE_NOTES.md Added

coveralls: 
[![Coverage Status](https://coveralls.io/builds/26831313/badge)](https://coveralls.io/builds/26831313)

Coverage decreased (-0.3%) to 85.04% when pulling **3707303aa3b08a0ce5054b1624685f19e97e4f9c on feature/query-region-synonyms** into **a4e266c93f531f94285311fd354aaa881aa403c4 on master**.

byersiiasa: lgtm
gidden: Haven't had a chance to review implementation, but we need at least one functionality test
zikolach: @gidden @danielhuppmann I added a test (as backend used for integration testing is now up-to-date). Can you please re-review?
danielhuppmann: thanks @zikolach - I was hoping that @gidden or @byersiiasa would review this, because I'm not sure how this is supposed to work...

Can you please rebase to resolve the merge conflict (changed `logger` implementation in #288)?

More generally, I'm not sure if this return-object (in wide format) is helpful, because one would need to check across an unknown number of columns (if I understand the format with `synonym_{}` correctly?). I think a return-object in long-format (see https://github.com/iiasa/ixmp/blob/399e1b04db32b8fb6f090dba611a900fd4c421ea/ixmp/core.py#L219) would be more practical?

Also, we should be able to tag "types" to synonyms, to distinguish between standard name, ISO3, alternative spelling, ...
zikolach: PR rebased

> More generally, I'm not sure if this return-object (in wide format) is helpful, because one would need to check across an unknown number of columns (if I understand the format with synonym_{} correctly?). I think a return-object in long-format (see https://github.com/iiasa/ixmp/blob/399e1b04db32b8fb6f090dba611a900fd4c421ea/ixmp/core.py#L219) would be more practical?

The intention was to keep it kind of backward-compatible (meaning regions method should return same (unique) list of regions). What do you think about adding/extending method parameters, e.g. instead of `include_synonyms` have `synonyms_format` with:
- default option (None) to omit synonyms
- option `columns` - current behavior
- option `rows` - long format

> Also, we should be able to tag "types" to synonyms, to distinguish between standard name, ISO3, alternative spelling, ...

Can you explain a bit more in detail what you mean? Dtypes? What is recommended way to do it?
danielhuppmann: Thanks @zikolach!

For backward compatibility, `include_synonyms=False` can be used as default.

I don't see a practical use case for `columns` format, because an a-priori unknown number of meaningful entries would make it a bit tedious working with a wide format. So just return `rows` (long format) for `include_synonyms=True`.

> > Also, we should be able to tag "types" to synonyms, to distinguish between standard name, ISO3, alternative spelling, ...

> Can you explain a bit more in detail what you mean? Dtypes? What is recommended way to do it?

Ignore this comment, let's tackle this at some future point - I realised this opens quite a few rabbit holes.
zikolach: > I don't see a practical use case for columns format, ...

Practical use case is to get first synonym for specific region. With long format it is a bit more difficult (as I can see from my limited pandas knowledge)
danielhuppmann: There is no such thing as a “first”, “second”, etc. synonym in the current logic. There can be any number of them, in any order.

We may want to introduce tags for synonyms, e.g. the ISO3 synonym (what I meant earlier). But this would require a major change in ixmp, restAPI, etc., so not to be tackled now.
zikolach: > There is no such thing as a “first”, “second”, etc.

I meant "first available synonym", but that is not important. Main constraint as mentioned was to keep region names unique in the result dataset independently on parameters. If that is no issue then it is ok from my side also to implement as you suggested. Please re-review changes.

Patch:
diff --git a/RELEASE_NOTES.md b/RELEASE_NOTES.md
index e95d1ea..2fc5cef 100644
--- a/RELEASE_NOTES.md
+++ b/RELEASE_NOTES.md
@@ -6,7 +6,7 @@
 - [#295](https://github.com/IAMconsortium/pyam/pull/295) Include `meta` table when writing to or reading from `xlsx` files
 - [#292](https://github.com/IAMconsortium/pyam/pull/292) Add warning message if `data` is empty at initialization (after formatting)
 - [#288](https://github.com/IAMconsortium/pyam/pull/288) Put `pyam` logger in its own namespace (see [here](https://docs.python-guide.org/writing/logging/#logging-in-a-library>))
-
+- [#285](https://github.com/IAMconsortium/pyam/pull/285) Add ability to fetch regions with synonyms from IXMP API
 # Release v0.3.0
 
 ## Highlights
diff --git a/pyam/iiasa.py b/pyam/iiasa.py
index d52015a..0cc71a0 100644
--- a/pyam/iiasa.py
+++ b/pyam/iiasa.py
@@ -235,13 +235,37 @@ class Connection(object):
         return pd.Series(df['variable'].unique(), name='variable')
 
     @lru_cache()
-    def regions(self):
-        """All regions in the connected data source"""
+    def regions(self, include_synonyms=False):
+        """All regions in the connected data source
+
+        :param include_synonyms: whether to include synonyms
+               (possibly leading to duplicate region names for
+               regions with more than one synonym)
+        """
         url = '/'.join([self._base_url, 'nodes?hierarchy=%2A'])
         headers = {'Authorization': 'Bearer {}'.format(self._token)}
-        r = requests.get(url, headers=headers)
+        params = {'includeSynonyms': include_synonyms}
+        r = requests.get(url, headers=headers, params=params)
         _check_response(r)
-        df = pd.read_json(r.content, orient='records')
+        return self.convert_regions_payload(r.content, include_synonyms)
+
+    @staticmethod
+    def convert_regions_payload(response, include_synonyms):
+        df = pd.read_json(response, orient='records')
+        if df.empty:
+            return df
+        if 'synonyms' not in df.columns:
+            df['synonyms'] = [list()] * len(df)
+        df = df.astype({
+            'id': str,
+            'name': str,
+            'hierarchy': str,
+            'parent': str,
+            'synonyms': object
+        })
+        if include_synonyms:
+            df = df[['name', 'synonyms']].explode('synonyms')
+            return df.rename(columns={'name': 'region', 'synonyms': 'synonym'})
         return pd.Series(df['name'].unique(), name='region')
 
     def _query_post_data(self, **kwargs):
@@ -330,7 +354,9 @@ class Connection(object):
         r = requests.post(url, headers=headers, data=data)
         _check_response(r)
         # refactor returned json object to be castable to an IamDataFrame
-        df = pd.read_json(r.content, orient='records')
+        dtype = dict(model=str, scenario=str, variable=str, unit=str,
+                     region=str, year=int, value=float, version=int)
+        df = pd.read_json(r.content, orient='records', dtype=dtype)
         logger.debug('Response size is {0} bytes, '
                        '{1} records'.format(len(r.content), len(df)))
         columns = ['model', 'scenario', 'variable', 'unit',


Test patch:
diff --git a/tests/test_iiasa.py b/tests/test_iiasa.py
index cbc2dd3..7335846 100644
--- a/tests/test_iiasa.py
+++ b/tests/test_iiasa.py
@@ -77,6 +77,50 @@ def test_regions():
     assert 'World' in obs
 
 
+def test_regions_with_synonyms():
+    conn = iiasa.Connection('IXSE_SR15')
+    obs = conn.regions(include_synonyms=True)
+    assert 'synonym' in obs.columns
+    assert (obs[obs.region == 'R5ROWO']
+            .synonym == 'Rest of the World (R5)').all()
+
+
+def test_regions_empty_response():
+    obs = iiasa.Connection.convert_regions_payload('[]', include_synonyms=True)
+    assert obs.empty
+
+
+def test_regions_no_synonyms_response():
+    json = '[{"id":1,"name":"World","parent":"World","hierarchy":"common"}]'
+    obs = iiasa.Connection.convert_regions_payload(json, include_synonyms=True)
+    assert not obs.empty
+
+
+def test_regions_with_synonyms_response():
+    json = '''
+    [
+        {
+            "id":1,"name":"World","parent":"World","hierarchy":"common",
+            "synonyms":[]
+        },
+        {
+            "id":2,"name":"USA","parent":"World","hierarchy":"country",
+            "synonyms":["US","United States"]
+        },
+        {
+            "id":3,"name":"Germany","parent":"World","hierarchy":"country",
+            "synonyms":["Deutschland","DE"]
+        }
+    ]
+    '''
+    obs = iiasa.Connection.convert_regions_payload(json, include_synonyms=True)
+    assert not obs.empty
+    assert (obs[obs.region == 'USA']
+            .synonym.isin(['US', 'United States'])).all()
+    assert (obs[obs.region == 'Germany']
+            .synonym.isin(['Deutschland', 'DE'])).all()
+
+
 def test_metadata():
     conn = iiasa.Connection('IXSE_SR15')
     obs = conn.scenario_list()['model'].values