Repository: IAMconsortium/pyam
Pull request #323 (2020-01-20 16:13:37)

add io-support for frictionless `datapackage`
# Please confirm that this PR has done the following:

- [x] Tests Added
- [x] Documentation Added
- [x] Description in RELEASE_NOTES.md Added

# Description of PR

This PR adds functions (and a unit test) to export and read (zipped) files in the frictionless datapackage format (https://frictionlessdata.io/). It's a first stab exporting/importing only the `data` and `meta` tables, not incorporating metadata (license) or other fancy features like consistency validation.


coveralls: 
[![Coverage Status](https://coveralls.io/builds/28214381/badge)](https://coveralls.io/builds/28214381)

Coverage decreased (-0.2%) to 85.609% when pulling **c20719c5d01520ff9648eb856ca3c0abcad4eaeb on danielhuppmann:io/datapackage** into **d6bd877f2822522021f58bdf16f531cc3860cb0a on IAMconsortium:master**.

danielhuppmann: Following the discussions at the openmod workshop, this PR implements a first stab at frictionless-datapackage-input/output of IAMC-like-timeseries data (plus categorisations and quantitative indicators).

@Ludee, @cmutel, @rgieseke [1], any comments? is anything critical missing?

[1] I played around with [pandas-datapackage-reader](https://github.com/rgieseke/pandas-datapackage-reader) at first, but the direct path using `datapackage` seemed easier in the end...
rgieseke: As Data Packages have become used in the energy modelling community this could be a very helpful feature. I'm still not sure whether a format with metadata in separate files (datapackage.json) is perfect for science (were files end up being mailed around), though the ZIP format might help with that.

In general i'm not sure whether the wide format of pyam (years on "x-axis") works nice with Data Package column based validation tools and metadata format tools, as this would require updating the metadata if a new year is added to the data. 
danielhuppmann: Thanks for your review comments @rgieseke!

> In general i'm not sure whether the wide format of pyam (years on "x-axis") works nice with Data Package column based validation tools and metadata format tools, as this would require updating the metadata if a new year is added to the data.

pyam works internally with a **long format** of timeseries data, and the implementation of `to_datapackage()` writes the csv-file in that way. This is in contrast to the function [IamDataFrame.to_csv()](https://pyam-iamc.readthedocs.io/en/latest/api.html#pyam.IamDataFrame.to_csv), which writes the more human-readable (and smaller if unzipped) **wide format**.

By the way, we spent some time on casting different table structures into an IAMC-compatible format, and this should pretty much work out-of-the-box:

 - long or wide
 - variables as columns
 - identifiers spread out over multiple columns (rather than the concatenated „variable“ column that we use)
 - a weird R „feature“ that adds an X to an integer column name when writing to file

No documentation yet, but a suite of [unit tests](https://github.com/IAMconsortium/pyam/blob/master/tests/test_cast_to_iamc.py)... If there is demand for a docs page of this, please make an issue!

rgieseke:     >   long or wide

Awesome, then there shouldn't be any problem with a Data Package i think (which could also easily be *read* by pandas-datapackage-reader ...
danielhuppmann: thanks @rgieseke and @cmutel for the helpful review comments!
danielhuppmann: @gidden, I found the relevant command in AppVeyor to install optional dependencies, but not for Travis CI (which is now failing because of the missing `datapackage`). Can you point to the right line I need to add?
danielhuppmann: ok, @gidden, seems the optional dependency now also works on Travis
danielhuppmann: now doing the optional-dependency install correctly via the makefile...

Patch:
diff --git a/.gitignore b/.gitignore
index 0d2e4c4..2445056 100644
--- a/.gitignore
+++ b/.gitignore
@@ -6,6 +6,8 @@
 *png
 *jpg
 *pdf
+*zip
+*json
 
 # Apple file system
 *.DS_STORE
diff --git a/Makefile b/Makefile
index 0c02b9c..5bea2f2 100644
--- a/Makefile
+++ b/Makefile
@@ -94,7 +94,7 @@ $(VENV_DIR):  $(CI_ENVIRONMENT_CONDA_DEFAULT_FILE) $(CI_ENVIRONMENT_CONDA_FORGE_
 	$(CONDA_EXE) config --add channels conda-forge # sets conda-forge as highest priority
 	$(CONDA_EXE) install --yes $(shell cat $(CI_ENVIRONMENT_CONDA_DEFAULT_FILE) $(CI_ENVIRONMENT_CONDA_FORGE_FILE) | tr '\n' ' ')
 	# Install development setup
-	$(VENV_DIR)/bin/pip install -e .[tests,deploy]
+	$(VENV_DIR)/bin/pip install -e .[tests,deploy,optional-io-formats]
 	# install docs requirements
 	# --name $(CONDA_DEFAULT_ENV) ensures we install in active environment (check at
 	# top of Makefile ensures that environment is not the base one)
diff --git a/RELEASE_NOTES.md b/RELEASE_NOTES.md
index c973549..53f3a04 100644
--- a/RELEASE_NOTES.md
+++ b/RELEASE_NOTES.md
@@ -1,5 +1,7 @@
 # Next release
 
+- [#323](https://github.com/IAMconsortium/pyam/pull/323) Support import/export of frictionless `datapackage` format
+
 # Release v0.4.0
 
 ## Highlights
diff --git a/appveyor.yml b/appveyor.yml
index 9005d10..17cdb5c 100644
--- a/appveyor.yml
+++ b/appveyor.yml
@@ -29,7 +29,7 @@ build: false
 
 test_script:
   - conda list
-  - pip install -e .[tests,deploy]
+  - pip install -e .[tests,deploy,optional-io-formats]
   - pytest -v --mpl tests --mpl-results-path=test-artifacts
 
 artifacts:
diff --git a/ci/environment-conda-forge.txt b/ci/environment-conda-forge.txt
index 48f6642..2751d40 100644
--- a/ci/environment-conda-forge.txt
+++ b/ci/environment-conda-forge.txt
@@ -4,7 +4,5 @@ matplotlib-base==3.0.3
 seaborn==0.9.0
 gdal
 fiona
-"geopandas<0.5.0"
 "poppler<0.66"
 cython
-cartopy
diff --git a/pyam/core.py b/pyam/core.py
index 00f10eb..49d2826 100755
--- a/pyam/core.py
+++ b/pyam/core.py
@@ -8,6 +8,15 @@ import sys
 import numpy as np
 import pandas as pd
 
+from pathlib import Path
+from tempfile import TemporaryDirectory
+
+try:
+    from datapackage import Package
+    HAS_DATAPACKAGE = True
+except ImportError:
+    HAS_DATAPACKAGE = False
+
 try:
     import ixmp
     has_ix = True
@@ -1284,6 +1293,39 @@ class IamDataFrame(object):
         if close:
             excel_writer.close()
 
+    def to_datapackage(self, path):
+        """Write object to a frictionless Data Package
+
+        More information: https://frictionlessdata.io
+
+        Returns the saved :class:`datapackage.Package`. When adding metadata
+        (descriptors), please follow the `template` defined by
+        https://github.com/OpenEnergyPlatform/metadata
+
+        Parameters
+        ----------
+        path: string or pathlib.Path
+            file path
+        """
+
+        if not HAS_DATAPACKAGE:
+            raise ImportError('required package `datapackage` not found!')
+
+        with TemporaryDirectory(dir='.') as tmp:
+            # save data and meta tables to a temporary folder
+            self.data.to_csv(Path(tmp) / 'data.csv', index=False)
+            self.meta.to_csv(Path(tmp) / 'meta.csv')
+
+            # cast tables to datapackage
+            package = Package()
+            package.infer('{}/*.csv'.format(tmp))
+            if not package.valid:
+                logger.warning('the exported package is not valid')
+            package.save(path)
+
+        # return the package (needs to reloaded because `tmp` was deleted)
+        return Package(path)
+
     def load_metadata(self, path, *args, **kwargs):
         """Deprecated, see :method:`load_meta()`"""
         # TODO: deprecate in next release (>=0.5.0)
@@ -1771,3 +1813,41 @@ def concat(dfs):
         else:
             _df.append(df, inplace=True)
     return _df
+
+
+def read_datapackage(path, data='data', meta='meta'):
+    """Read timeseries data and meta-indicators from frictionless Data Package
+
+    Parameters
+    ----------
+    path: path to Data Package
+        passed to ``datapackage.Package()``
+    data: str, default `data`
+        resource containing timeseries data in IAMC-compatible format
+    meta: str, default `meta`
+        (optional) resource containing a table of categorization and
+        quantitative indicators
+    """
+
+    if not HAS_DATAPACKAGE:
+        raise ImportError('required package `datapackage` not found!')
+
+    package = Package(path)
+
+    def _get_column_names(x):
+        return [i['name'] for i in x.descriptor['schema']['fields']]
+
+    # read `data` table
+    resource_data = package.get_resource(data)
+    _data = pd.DataFrame(resource_data.read())
+    _data.columns = _get_column_names(resource_data)
+    df = IamDataFrame(_data)
+
+    # read `meta` table
+    if meta in package.resource_names:
+        resource_meta = package.get_resource(meta)
+        _meta = pd.DataFrame(resource_meta.read())
+        _meta.columns = _get_column_names(resource_meta)
+        df.meta = _meta.set_index(META_IDX)
+
+    return df
diff --git a/setup.py b/setup.py
index be5e9f1..68ce727 100755
--- a/setup.py
+++ b/setup.py
@@ -22,7 +22,7 @@ REQUIREMENTS = [
     'argparse',
     'numpy',
     'requests',
-    'pandas>=0.25.0',
+    'pandas>=0.25.0,<1.0.0',
     'PyYAML',
     'xlrd',
     'xlsxwriter',
@@ -33,6 +33,7 @@ REQUIREMENTS = [
 
 EXTRA_REQUIREMENTS = {
     'tests': ['coverage', 'coveralls', 'pytest', 'pytest-cov', 'pytest-mpl'],
+    'optional-io-formats': ['datapackage'],
     'deploy': ['twine', 'setuptools', 'wheel'],
 }
 


Test patch:
diff --git a/pyam/testing.py b/pyam/testing.py
index 9a3699d..941d15a 100644
--- a/pyam/testing.py
+++ b/pyam/testing.py
@@ -2,7 +2,7 @@ from . import compare
 import pandas.testing as pdt
 
 
-def assert_frame_equal(a, b, **assert_kwargs):
+def assert_iamframe_equal(a, b, **assert_kwargs):
     diff = compare(a, b)
     if not diff.empty:
         msg = 'IamDataFrame.data are different: \n {}'
diff --git a/tests/test_feature_aggregate.py b/tests/test_feature_aggregate.py
index ec3f85d..2d0ef2b 100644
--- a/tests/test_feature_aggregate.py
+++ b/tests/test_feature_aggregate.py
@@ -4,7 +4,7 @@ import logging
 import numpy as np
 import pandas as pd
 from pyam import check_aggregate, IamDataFrame, IAMC_IDX
-from pyam.testing import assert_frame_equal
+from pyam.testing import assert_iamframe_equal
 
 from conftest import DTS_MAPPING
 
@@ -48,7 +48,7 @@ PRICE_MAX_DF = pd.DataFrame([
 def test_aggregate(simple_df, variable, data):
     # check that `variable` is a a direct sum and matches given total
     exp = simple_df.filter(variable=variable)
-    assert_frame_equal(simple_df.aggregate(variable), exp)
+    assert_iamframe_equal(simple_df.aggregate(variable), exp)
 
     # use other method (max) both as string and passing the function
     _df = data.copy()
@@ -57,7 +57,7 @@ def test_aggregate(simple_df, variable, data):
         _df.rename({'year': 'time'}, axis='columns', inplace=True)
     exp = IamDataFrame(_df)
     for m in ['max', np.max]:
-        assert_frame_equal(simple_df.aggregate(variable, method=m), exp)
+        assert_iamframe_equal(simple_df.aggregate(variable, method=m), exp)
 
 
 def test_check_aggregate(simple_df):
@@ -103,7 +103,7 @@ def test_aggregate_append(simple_df, variable):
     # remove `variable`, do aggregate and append, check equality to original
     _df = simple_df.filter(variable=variable, keep=False)
     _df.aggregate(variable, append=True)
-    assert_frame_equal(_df, simple_df)
+    assert_iamframe_equal(_df, simple_df)
 
 
 def test_aggregate_with_components(simple_df):
@@ -138,12 +138,13 @@ def test_aggregate_unknown_method(simple_df):
 def test_aggregate_region(simple_df, variable):
     # check that `variable` is a a direct sum across regions
     exp = simple_df.filter(variable=variable, region='World')
-    assert_frame_equal(simple_df.aggregate_region(variable), exp)
+    assert_iamframe_equal(simple_df.aggregate_region(variable), exp)
 
     # check custom `region` (will include `World`, so double-count values)
     foo = exp.rename(region={'World': 'foo'})
     foo.data.value = foo.data.value * 2
-    assert_frame_equal(simple_df.aggregate_region(variable, region='foo'), foo)
+    assert_iamframe_equal(simple_df.aggregate_region(variable, region='foo'),
+                          foo)
 
 
 def test_aggregate_region_log(simple_df, caplog):
@@ -191,7 +192,7 @@ def test_aggregate_region_append(simple_df, variable):
     # remove `variable`, do aggregate and append, check equality to original
     _df = simple_df.filter(variable=variable, region='World', keep=False)
     _df.aggregate_region(variable, append=True)
-    assert_frame_equal(_df, simple_df)
+    assert_iamframe_equal(_df, simple_df)
 
 
 @pytest.mark.parametrize("variable", (
@@ -205,13 +206,13 @@ def test_aggregate_region_with_subregions(simple_df, variable):
         .rename(region={'reg_a': 'World'})
     )
     obs = simple_df.aggregate_region(variable, subregions='reg_a')
-    assert_frame_equal(obs, exp)
+    assert_iamframe_equal(obs, exp)
 
     # check that both custom `region` and `subregions` work
     foo = exp.rename(region={'World': 'foo'})
     obs = simple_df.aggregate_region(variable, region='foo',
                                      subregions='reg_a')
-    assert_frame_equal(obs, foo)
+    assert_iamframe_equal(obs, foo)
 
     # check that invalid list of subregions returns empty
     assert simple_df.aggregate_region(variable, subregions=['reg_c']).empty
@@ -229,7 +230,8 @@ def test_aggregate_region_with_other_method(simple_df, variable, data):
         _df.rename({'year': 'time'}, axis='columns', inplace=True)
     exp = IamDataFrame(_df).filter(region='World')
     for m in ['max', np.max]:
-        assert_frame_equal(simple_df.aggregate_region(variable, method=m), exp)
+        assert_iamframe_equal(simple_df.aggregate_region(variable, method=m),
+                              exp)
 
 
 def test_aggregate_region_with_components(simple_df):
@@ -251,7 +253,7 @@ def test_aggregate_region_with_weights(simple_df):
     assert simple_df.check_aggregate_region(v, weight=w) is None
 
     exp = simple_df.filter(variable=v, region='World')
-    assert_frame_equal(simple_df.aggregate_region(v, weight=w), exp)
+    assert_iamframe_equal(simple_df.aggregate_region(v, weight=w), exp)
 
     # inconsistent index of variable and weight raises an error
     _df = simple_df.filter(variable=w, region='reg_b', keep=False)
diff --git a/tests/test_feature_downscale.py b/tests/test_feature_downscale.py
index 9509830..5fdb18f 100644
--- a/tests/test_feature_downscale.py
+++ b/tests/test_feature_downscale.py
@@ -1,5 +1,5 @@
 import pytest
-from pyam.testing import assert_frame_equal
+from pyam.testing import assert_iamframe_equal
 
 
 @pytest.mark.parametrize("variable", (
@@ -13,9 +13,9 @@ def test_downscale_region(simple_df, variable):
     # return as new IamDataFrame
     obs = simple_df.downscale_region(variable, proxy='Population')
     exp = simple_df.filter(variable=variable, region=regions)
-    assert_frame_equal(exp, obs)
+    assert_iamframe_equal(exp, obs)
 
     # append to `self` (after removing to-be-downscaled timeseries)
     inplace = simple_df.filter(variable=variable, region=regions, keep=False)
     inplace.downscale_region(variable, proxy='Population', append=True)
-    assert_frame_equal(inplace, simple_df)
+    assert_iamframe_equal(inplace, simple_df)
diff --git a/tests/test_io.py b/tests/test_io.py
index 89ac744..e8380fd 100644
--- a/tests/test_io.py
+++ b/tests/test_io.py
@@ -3,7 +3,8 @@ import pandas as pd
 import numpy as np
 import pytest
 
-from pyam import IamDataFrame
+from pyam import IamDataFrame, read_datapackage
+from pyam.testing import assert_iamframe_equal
 
 from conftest import TEST_DATA_DIR
 
@@ -41,9 +42,8 @@ def test_io_xlsx(test_df, meta_args):
         # read from xlsx
         import_df = IamDataFrame(file, **meta_args[1])
 
-        # assert that `data` and `meta` tables are equal and delete file
-        pd.testing.assert_frame_equal(test_df.data, import_df.data)
-        pd.testing.assert_frame_equal(test_df.meta, import_df.meta)
+        # assert that IamDataFrame instances are equal and delete file
+        assert_iamframe_equal(test_df, import_df)
         os.remove(file)
 
 
@@ -74,3 +74,20 @@ def test_load_rcp_database_downloaded_file(test_df_year):
         TEST_DATA_DIR, 'test_RCP_database_raw_download.xlsx')
     )
     pd.testing.assert_frame_equal(obs_df.as_pandas(), exp)
+
+
+def test_io_datapackage(test_df):
+    file = 'foo.zip'
+
+    # add column to `meta`
+    test_df.set_meta(['a', 'b'], 'string')
+
+    # write to datapackage
+    test_df.to_datapackage(file)
+
+    # read from csv
+    import_df = read_datapackage(file)
+
+    # assert that IamDataFrame instances are equal and delete file
+    assert_iamframe_equal(test_df, import_df)
+    os.remove(file)