Repository: IAMconsortium/pyam
Pull request #486 (2021-01-22 09:09:29)

Enable reading non-default scenario versions from IIASA API
# Please confirm that this PR has done the following:

- [x] Tests Added
- [x] Documentation Added
- ~Name of contributors Added to AUTHORS.rst~
- [x] Description in RELEASE_NOTES.md Added

# Description of PR

This PR uses the "custom-index" feature (currently in beta support, see #481) to read data of non-default scenario versions from the IIASA RestAPI and return an IamDataFrame with a non-standard index.


danielhuppmann: @benke-pik, you may like this new feature... Try `pyam.read_iiasa(<instance>, default=False)` to get not just the latest version of a model-scenario but all of versions uploaded to the IIASA database...
codecov[bot]: # [Codecov](https://codecov.io/gh/IAMconsortium/pyam/pull/486?src=pr&el=h1) Report
> Merging [#486](https://codecov.io/gh/IAMconsortium/pyam/pull/486?src=pr&el=desc) (698b7af) into [master](https://codecov.io/gh/IAMconsortium/pyam/commit/3d31fdd36f9b1ca6088eadbc01d23857b9fab4ae?el=desc) (3d31fdd) will **increase** coverage by `0.03%`.
> The diff coverage is `100.00%`.

[![Impacted file tree graph](https://codecov.io/gh/IAMconsortium/pyam/pull/486/graphs/tree.svg?width=650&height=150&src=pr&token=ThxWlNkonJ)](https://codecov.io/gh/IAMconsortium/pyam/pull/486?src=pr&el=tree)

```diff
@@            Coverage Diff             @@
##           master     #486      +/-   ##
==========================================
+ Coverage   92.96%   92.99%   +0.03%     
==========================================
  Files          41       41              
  Lines        4746     4756      +10     
==========================================
+ Hits         4412     4423      +11     
+ Misses        334      333       -1     
```


| [Impacted Files](https://codecov.io/gh/IAMconsortium/pyam/pull/486?src=pr&el=tree) | Coverage Δ | |
|---|---|---|
| [pyam/iiasa.py](https://codecov.io/gh/IAMconsortium/pyam/pull/486/diff?src=pr&el=tree#diff-cHlhbS9paWFzYS5weQ==) | `89.18% <100.00%> (+0.30%)` | :arrow_up: |
| [tests/test\_iiasa.py](https://codecov.io/gh/IAMconsortium/pyam/pull/486/diff?src=pr&el=tree#diff-dGVzdHMvdGVzdF9paWFzYS5weQ==) | `92.80% <100.00%> (+0.67%)` | :arrow_up: |

------

[Continue to review full report at Codecov](https://codecov.io/gh/IAMconsortium/pyam/pull/486?src=pr&el=continue).
> **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta)
> `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
> Powered by [Codecov](https://codecov.io/gh/IAMconsortium/pyam/pull/486?src=pr&el=footer). Last update [3d31fdd...698b7af](https://codecov.io/gh/IAMconsortium/pyam/pull/486?src=pr&el=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments).


Patch:
diff --git a/RELEASE_NOTES.md b/RELEASE_NOTES.md
index 89579d2..5b9a360 100644
--- a/RELEASE_NOTES.md
+++ b/RELEASE_NOTES.md
@@ -8,8 +8,11 @@ starting with `data` will be parsed for timeseries data.
 
 ## Individual updates
 
+- [#496](https://github.com/IAMconsortium/pyam/pull/496) Enable loading meta from csv file
+- [#494](https://github.com/IAMconsortium/pyam/pull/494) Small performance improvements
 - [#491](https://github.com/IAMconsortium/pyam/pull/491) Allow loading meta from file without exclude columns or with empty rows
 - [#488](https://github.com/IAMconsortium/pyam/pull/488) Read all sheets starting with `data` when reading from xlsx
+- [#486](https://github.com/IAMconsortium/pyam/pull/486) Enable reading non-default scenario versions from IIASA API
 - [#481](https://github.com/IAMconsortium/pyam/pull/481) Enable custom index columns
 - [#477](https://github.com/IAMconsortium/pyam/pull/477) Add a nightly test suite
 - [#476](https://github.com/IAMconsortium/pyam/pull/476) Add docstrings to plotting functions `df.plot.<kind>()`
diff --git a/pyam/core.py b/pyam/core.py
index 4451a27..af448f2 100755
--- a/pyam/core.py
+++ b/pyam/core.py
@@ -229,7 +229,7 @@ class IamDataFrame(object):
             self.data = df
 
     def __len__(self):
-        return self.data.__len__()
+        return len(self._data)
 
     def __repr__(self):
         return self.info()
@@ -1731,17 +1731,18 @@ class IamDataFrame(object):
 
         Parameters
         ----------
-        path : str or :class:`pathlib.Path`
-            A valid path
+        path : str, :class:`pathlib.Path` or :class:`pandas.ExcelFile`
+            A valid path or instance of an xlsx or csv file
         sheet_name : str, optional
-            Name of the sheet to be parsed
+            Name of the sheet to be parsed (if xlsx)
         ignore_conflict : bool, optional
             If `True`, values in `path` take precedence over existing `meta`.
             If `False`, raise an error in case of conflicts.
         kwargs
-            passed to :func:`pandas.read_excel`
+            Passed to :func:`pandas.read_excel` or :func:`pandas.read_csv`
         """
         # load from file
+        path = path if isinstance(path, pd.ExcelFile) else Path(path)
         df = read_pandas(path, sheet_name=sheet_name, **kwargs)
 
         # cast model-scenario column headers to lower-case (if necessary)
diff --git a/pyam/iiasa.py b/pyam/iiasa.py
index 4e50678..7a8999e 100644
--- a/pyam/iiasa.py
+++ b/pyam/iiasa.py
@@ -11,7 +11,8 @@ import pandas as pd
 
 from collections.abc import Mapping
 from pyam.core import IamDataFrame
-from pyam.utils import META_IDX, IAMC_IDX, isstr, pattern_match
+from pyam.utils import META_IDX, IAMC_IDX, isstr, pattern_match, \
+    DEFAULT_META_INDEX, islistable
 from pyam.logging import deprecation_warning
 
 logger = logging.getLogger(__name__)
@@ -225,26 +226,18 @@ class Connection(object):
         return self._query_index(default)[META_IDX + cols].set_index(META_IDX)
 
     @lru_cache()
-    def _query_index(self, default=True):
-        # TODO merge this function with `meta()`
-        default = 'true' if default else 'false'
-        add_url = 'runs?getOnlyDefaultRuns={}'
-        url = '/'.join([self._auth_url, add_url.format(default)])
-        headers = {'Authorization': 'Bearer {}'.format(self._token)}
-        r = requests.get(url, headers=headers)
-        _check_response(r, 'Could not retrieve the resource index')
-        return pd.read_json(r.content, orient='records')
-
-    @lru_cache()
-    def _query_meta(self, default=True):
+    def _query_index(self, default=True, meta=False):
         # TODO: at present this reads in all data for all scenarios,
         #  it could be sped up in the future to try to query a subset
         _default = 'true' if default else 'false'
-        add_url = 'runs?getOnlyDefaultRuns={}&includeMetadata=true'
-        url = '/'.join([self._auth_url, add_url.format(_default)])
+        _meta = 'true' if meta else 'false'
+        add_url = f'runs?getOnlyDefaultRuns={_default}&includeMetadata={_meta}'
+        url = '/'.join([self._auth_url, add_url])
         headers = {'Authorization': 'Bearer {}'.format(self._token)}
         r = requests.get(url, headers=headers)
         _check_response(r)
+
+        # cast response to dataframe and return
         return pd.read_json(r.content, orient='records')
 
     @property
@@ -257,8 +250,7 @@ class Connection(object):
         _check_response(r)
         return pd.read_json(r.content, orient='records')['name']
 
-    @lru_cache()
-    def meta(self, default=True):
+    def meta(self, default=True, **kwargs):
         """Return categories and indicators (meta) of scenarios
 
         Parameters
@@ -266,10 +258,14 @@ class Connection(object):
         default : bool, optional
             Return *only* the default version of each scenario.
             Any (`model`, `scenario`) without a default version is omitted.
-            If :obj:`False`, return all versions.
+            If `False`, return all versions.
         """
-        df = self._query_meta(default)
+        df = self._query_index(default, meta=True)
         cols = ['version'] if default else ['version', 'is_default']
+        if kwargs:
+            if kwargs.pop('run_id', False):
+                cols += ['run_id']
+        index = DEFAULT_META_INDEX + ([] if default else ['version'])
 
         def extract(row):
             return (
@@ -277,7 +273,7 @@ class Connection(object):
                            pd.Series(row.metadata)])
                 .to_frame()
                 .T
-                .set_index(['model', 'scenario'])
+                .set_index(index)
             )
 
         return pd.concat([extract(row) for i, row in df.iterrows()],
@@ -293,7 +289,7 @@ class Connection(object):
             Any (`model`, `scenario`) without a default version is omitted.
             If :obj:`False`, return all versions.
         """
-        _df = self._query_meta(default)
+        _df = self._query_index(default, meta=True)
         audit_cols = ['cre_user', 'cre_date', 'upd_user', 'upd_date']
         audit_mapping = dict([(i, i.replace('_', 'ate_')) for i in audit_cols])
         other_cols = ['version'] if default else ['version', 'is_default']
@@ -361,7 +357,7 @@ class Connection(object):
             return df.rename(columns={'name': 'region', 'synonyms': 'synonym'})
         return pd.Series(df['name'].unique(), name='region')
 
-    def _query_post_data(self, **kwargs):
+    def _query_post(self, meta, default=True, **kwargs):
         def _get_kwarg(k):
             # TODO refactor API to return all models if model-list is empty
             x = kwargs.pop(k, '*' if k == 'model' else [])
@@ -382,9 +378,12 @@ class Connection(object):
                 matches |= pattern_match(data, p)
             return data[matches].unique()
 
-        # get unique run ids
-        meta = self._query_index()
-        meta = meta[meta.is_default]
+        # drop non-default runs if only default is requested
+        if default and hasattr(meta, 'is_default'):
+            meta = meta[meta.is_default]
+
+        # determine relevant run id's
+        meta = meta.reset_index()
         models = _match(meta['model'], m_pattern)
         scenarios = _match(meta['scenario'], s_pattern)
         if len(models) == 0 and len(scenarios) == 0:
@@ -455,17 +454,25 @@ class Connection(object):
                              variable=['Emissions|CO2', 'Primary Energy'])
 
         """
-        # TODO: API returns timeseries data for non-default versions
-        if default is not True:
-            msg = 'Querying for non-default scenarios is not (yet) supported'
-            raise ValueError(msg)
-
-        # retrieve data
         headers = {
             'Authorization': 'Bearer {}'.format(self._token),
             'Content-Type': 'application/json',
         }
-        _args = json.dumps(self._query_post_data(**kwargs))
+
+        # retrieve meta (with run ids) or only index
+        if meta:
+            _meta = self.meta(default=default, run_id=True)
+            # downselect to subset of meta columns if given as list
+            if islistable(meta):
+                # always merge 'version' (even if not requested explicitly)
+                # 'run_id' is required to determine `_args`, dropped later
+                _meta = _meta[set(meta).union(['version', 'run_id'])]
+        else:
+            _meta = self._query_index(default=default)\
+                .set_index(DEFAULT_META_INDEX)
+
+        # retrieve data
+        _args = json.dumps(self._query_post(_meta, default=default, **kwargs))
         url = '/'.join([self._auth_url, 'runs/bulk/ts'])
         logger.debug(f'Query timeseries data from {url} with data {_args}')
         r = requests.post(url, headers=headers, data=_args)
@@ -485,30 +492,23 @@ class Connection(object):
             if all([i in [-1, 'Year'] for i in timeslices]):
                 data.drop(columns='subannual', inplace=True)
 
-        # check if there are multiple version for any model/scenario
-        lst = (
-            data[META_IDX + ['version']].drop_duplicates()
-            .groupby(META_IDX).count().version
-        )
-
-        # checking if there are multiple versions
-        # for every model/scenario combination
-        # TODO this is probably not necessary
-        if len(lst) > 1 and max(lst) > 1:
-            raise ValueError('multiple versions for {}'.format(
-                lst[lst > 1].index.to_list()))
-        data.drop(columns='version', inplace=True)
-
-        # cast to IamDataFrame
-        df = IamDataFrame(data)
+        # define the index for the IamDataFrame
+        if default:
+            index = DEFAULT_META_INDEX
+            data.drop(columns='version', inplace=True)
+        else:
+            index = DEFAULT_META_INDEX + ['version']
+            logger.info('Initializing an `IamDataFrame` '
+                        f'with non-default index {index}')
 
-        # merge meta categorization and quantitative indications
+        # merge meta indicators (if requested) and cast to IamDataFrame
         if meta:
-            _meta = self.meta().loc[df.meta.index]
-            for i in _meta.columns if meta is True else meta + ['version']:
-                df.set_meta(_meta[i])
-
-        return IamDataFrame(df)
+            # 'run_id' is necessary to retrieve data, not returned by default
+            if not (islistable(meta) and 'run_id' in meta):
+                _meta.drop(columns='run_id', inplace=True)
+            return IamDataFrame(data, meta=_meta, index=index)
+        else:
+            return IamDataFrame(data, index=index)
 
 
 def read_iiasa(name, default=True, meta=True, creds=None, base_url=_AUTH_URL,
diff --git a/pyam/index.py b/pyam/index.py
index 34ac86b..926f6b3 100644
--- a/pyam/index.py
+++ b/pyam/index.py
@@ -1,4 +1,5 @@
 import pandas as pd
+import numpy as np
 
 
 def get_index_levels(df, level):
@@ -8,21 +9,21 @@ def get_index_levels(df, level):
 
 def replace_index_values(df, level, mapping):
     """Replace one or several category-values at a specific level"""
-    index = df.index.copy()
+    index = df if isinstance(df, pd.Index) else df.index
+
     n = index._get_level_number(level)
 
     # replace the levels
-    _levels = [mapping[i] if i in mapping else i for i in index.levels[n]]
-    _unique_levels = list(set(_levels))
+    _levels = index.levels[n].map(lambda l: mapping.get(l, l))
+    _unique_levels = _levels.unique()
 
     # if no duplicate levels exist after replace, set new levels and return
     if len(_levels) == len(_unique_levels):
         return index.set_levels(_levels, n)
 
     # if duplicate levels exist, re-map the codes
-    levels_mapping = dict([(i, _unique_levels.index(lvl))
-                           for i, lvl in enumerate(_levels)])
-    _codes = [levels_mapping[i] for i in index.codes[n]]
+    level_mapping = _unique_levels.get_indexer(_levels)
+    _codes = np.where(index.codes[n] != -1, level_mapping[index.codes[n]], -1)
     return index.set_codes(_codes, n).set_levels(_unique_levels, n)
 
 
diff --git a/pyam/units.py b/pyam/units.py
index 3d50fd1..62547f6 100644
--- a/pyam/units.py
+++ b/pyam/units.py
@@ -25,7 +25,7 @@ def convert_unit(df, current, to, factor=None, registry=None, context=None,
 
     if factor:
         # Short code path: use an explicit conversion factor, don't use pint
-        ret._data[where] *= factor
+        ret._data.loc[where] *= factor
         ret._data.index = replace_index_values(*index_args)
         return None if inplace else ret
 
@@ -51,7 +51,7 @@ def convert_unit(df, current, to, factor=None, registry=None, context=None,
         raise TypeError(f'{registry} must be `pint.UnitRegistry`') from None
 
     # Copy values from the result Quantity and assign units
-    ret._data[where] = result.magnitude
+    ret._data.loc[where] = result.magnitude
     ret._data.index = replace_index_values(*index_args)
 
     return None if inplace else ret


Test patch:
diff --git a/tests/test_iiasa.py b/tests/test_iiasa.py
index 7c84ee2..99de8a1 100644
--- a/tests/test_iiasa.py
+++ b/tests/test_iiasa.py
@@ -36,6 +36,15 @@ MODEL_B_DF = pd.DataFrame([
     ['Primary Energy|Coal', 'EJ/yr', 'Year', 0.9, 5]
 ], columns=['variable', 'unit', 'subannual', 2005, 2010])
 
+NON_DEFAULT_DF = pd.DataFrame([
+    ['model_a', 'scen_a', 2, 'Primary Energy', 'EJ/yr', 'Year', 2, 7],
+    ['model_a', 'scen_a', 2, 'Primary Energy|Coal', 'EJ/yr', 'Year', 0.8, 4],
+    ['model_b', 'scen_a', 1, 'Primary Energy', 'EJ/yr', 'Summer', 1, 3],
+    ['model_b', 'scen_a', 1, 'Primary Energy', 'EJ/yr', 'Year', 3, 8],
+    ['model_b', 'scen_a', 1, 'Primary Energy|Coal', 'EJ/yr', 'Summer', 0.4, 2],
+    ['model_b', 'scen_a', 1, 'Primary Energy|Coal', 'EJ/yr', 'Year', 0.9, 5]
+], columns=META_IDX + ['version', 'variable', 'unit', 'subannual', 2005, 2010])
+
 
 def test_unknown_conn():
     # connecting to an unknown API raises an error
@@ -158,10 +167,11 @@ def test_index(conn, default):
 @pytest.mark.parametrize("default", [True, False])
 def test_meta(conn, default):
     # test that connection returns the correct meta dataframe
+    v = 'version'
     if default:
-        exp = META_DF.loc[META_DF.is_default, ['version'] + META_COLS]
+        exp = META_DF.loc[META_DF.is_default, [v] + META_COLS]
     else:
-        exp = META_DF[VERSION_COLS + META_COLS]
+        exp = META_DF[VERSION_COLS + META_COLS].set_index(v, append=True)
 
     pdt.assert_frame_equal(conn.meta(default=default), exp, check_dtype=False)
 
@@ -225,12 +235,16 @@ def test_query_with_subannual(conn, test_pd_df, kwargs):
     assert_iamframe_equal(df, exp.filter(**kwargs))
 
 
+@pytest.mark.parametrize("meta", [
+    ['string'],  # version column is added whether or not stated explicitly
+    ['string', 'version']
+])
 @pytest.mark.parametrize("kwargs", [
     {},
     dict(variable='Primary Energy'),
     dict(scenario='scen_a', variable='Primary Energy')
 ])
-def test_query_with_meta_arg(conn, test_pd_df, kwargs):
+def test_query_with_meta_arg(conn, test_pd_df, meta, kwargs):
     # test reading timeseries data (including subannual data)
     exp = IamDataFrame(test_pd_df, subannual='Year')\
         .append(MODEL_B_DF, model='model_b', scenario='scen_a', region='World')
@@ -238,11 +252,11 @@ def test_query_with_meta_arg(conn, test_pd_df, kwargs):
         exp.set_meta(META_DF.iloc[[0, 1, 3]][i])
 
     # test method via Connection
-    df = conn.query(meta=['string'], **kwargs)
+    df = conn.query(meta=meta, **kwargs)
     assert_iamframe_equal(df, exp.filter(**kwargs))
 
     # test top-level method
-    df = read_iiasa(TEST_API, meta=['string'], **kwargs)
+    df = read_iiasa(TEST_API, meta=meta, **kwargs)
     assert_iamframe_equal(df, exp.filter(**kwargs))
 
 
@@ -265,6 +279,20 @@ def test_query_with_meta_false(conn, test_pd_df, kwargs):
     assert_iamframe_equal(df, exp.filter(**kwargs))
 
 
-def test_query_non_default(conn):
-    # querying for non-default scenario data raises an error
-    pytest.raises(ValueError, conn.query, default=False)
+def test_query_non_default(conn, test_pd_df):
+    # test reading timeseries data with non-default versions & index
+    test_pd_df['subannual'] = 'Year'
+    test_pd_df['version'] = 1
+    df = pd.concat([test_pd_df[NON_DEFAULT_DF.columns], NON_DEFAULT_DF])
+
+    meta = META_DF.set_index('version', append=True)
+    index = ['model', 'scenario', 'version']
+    exp = IamDataFrame(df, meta=meta, index=index, region='World')
+
+    # test method via Connection
+    df = conn.query(default=False)
+    assert_iamframe_equal(df, exp)
+
+    # test top-level method
+    df = read_iiasa(TEST_API, default=False)
+    assert_iamframe_equal(df, exp)
diff --git a/tests/test_io.py b/tests/test_io.py
index eb532ff..8b916e7 100644
--- a/tests/test_io.py
+++ b/tests/test_io.py
@@ -115,12 +115,10 @@ def test_init_df_with_na_unit(test_pd_df, tmpdir):
     ('foo', dict(sheet_name='foo'), False),
     ('foo', dict(sheet_name='foo'), True),
 ])
-def test_load_meta(test_pd_df, sheet_name, init_args, rename, tmpdir):
+def test_load_meta_xlsx(test_pd_df, sheet_name, init_args, rename, tmpdir):
     """Test loading meta from an Excel file"""
-    meta = META_DF.copy()
     # downselect meta
-    if rename:
-        meta = meta.iloc[0:1]
+    meta = META_DF.iloc[0:1] if rename else META_DF
 
     # initialize a new IamDataFrame directly from data and meta
     exp = IamDataFrame(test_pd_df, meta=meta)
@@ -136,6 +134,25 @@ def test_load_meta(test_pd_df, sheet_name, init_args, rename, tmpdir):
     assert_iamframe_equal(obs, exp)
 
 
+@pytest.mark.parametrize("rename", [True, False])
+def test_load_meta_csv(test_pd_df, rename, tmpdir):
+    """Test loading meta from an csv file"""
+    meta = META_DF.iloc[0:1] if rename else META_DF
+
+    # initialize a new IamDataFrame directly from data and meta
+    exp = IamDataFrame(test_pd_df, meta=meta)
+
+    # write meta to file (without an exclude col)
+    file = tmpdir / 'testing_io_meta.csv'
+    meta.reset_index().to_csv(file, index=False)
+
+    # initialize a new IamDataFrame and load meta from file
+    obs = IamDataFrame(test_pd_df)
+    obs.load_meta(file)
+
+    assert_iamframe_equal(obs, exp)
+
+
 def test_load_meta_wrong_index(test_df_year, tmpdir):
     """Loading meta without (at least) index cols as headers raises an error"""