Repository: IAMconsortium/pyam
Pull request #212 (2019-04-02 09:11:33)

R style columns
# Please confirm that this PR has done the following:

- [x] Tests Added
- [x] Documentation Added
- [x] Description in RELEASE_NOTES.md Added

# Description of PR

Who else has gotten very useful data from very useful colleages in that very pesky R-style data format, where year columns look like `X2015` instead of just `2015`? Worry no more, `pyam` has your back!


gidden: Mind taking a look @znicholls? I want to give @danielhuppmann a bit of a break from reviews (at least for one!).
gidden: Ok, can do. Errors in CI are not related, FYI.
znicholls: > Errors in CI are not related, FYI.

I guessed as much. Do you want me to split my fix from #198 out into a separate PR to see if that helps?
znicholls: > Do you want me to split my fix from #198 out into a separate PR to see if that helps?

Ignore that, just looked at the errors properly
coveralls: 
[![Coverage Status](https://coveralls.io/builds/22548061/badge)](https://coveralls.io/builds/22548061)

Coverage decreased (-0.0008%) to 84.74% when pulling **7ac3bb14557e03e812fd0369b01d94967a0f8ab6 on gidden:r-style-columns** into **02e2e0122f1b30ea64d8802ba2ea616eb48e7348 on IAMconsortium:master**.

gidden: tests will fail again because our backends are down from public access..
gidden: should be good to go now
gidden: once issue discussion resolved with @danielhuppmann 

Patch:
diff --git a/RELEASE_NOTES.md b/RELEASE_NOTES.md
index 2607e0b..3423ff2 100644
--- a/RELEASE_NOTES.md
+++ b/RELEASE_NOTES.md
@@ -1,6 +1,7 @@
 
 # Next Release
 
+- [#212](https://github.com/IAMconsortium/pyam/pull/212) Now natively support reading R-style data frames with year columns like "X2015"
 - [#202](https://github.com/IAMconsortium/pyam/pull/202) Extend the `df.rename()` function with a `check_duplicates (default True)` validation option
 - [#201](https://github.com/IAMconsortium/pyam/pull/201) Added native support for legends outside of plots with `pyam.plotting.OUTSIDE_LEGEND` with a tutorial
 - [#199](https://github.com/IAMconsortium/pyam/pull/199) Initializing an `IamDataFrame` accepts kwargs to fill or create from the data any missing required columns
diff --git a/pyam/core.py b/pyam/core.py
index 2bd170b..42a9b0e 100644
--- a/pyam/core.py
+++ b/pyam/core.py
@@ -57,6 +57,8 @@ class IamDataFrame(object):
         an instance of an TimeSeries or Scenario (requires `ixmp`),
         or pd.DataFrame or data file with IAMC-format data columns.
         A pd.DataFrame can have the required data as columns or index.
+        Support is provided additionally for R-style data columns for years,
+        like "X2015", etc.
     kwargs:
         if `value=col`, melt `col` to `value` and use `col` name as `variable`;
         else, mapping of columns required for an `IamDataFrame` to:
@@ -64,6 +66,7 @@ class IamDataFrame(object):
         - multiple columns, which will be concatenated by pipe
         - a string to be used as value for this column
     """
+
     def __init__(self, data, **kwargs):
         """Initialize an instance of an IamDataFrame"""
         # import data from pd.DataFrame or read from source
diff --git a/pyam/utils.py b/pyam/utils.py
index 5695495..8135fb2 100644
--- a/pyam/utils.py
+++ b/pyam/utils.py
@@ -132,6 +132,24 @@ def format_data(df, **kwargs):
     if isinstance(df, pd.Series):
         df = df.to_frame()
 
+    # Check for R-style year columns, converting where necessary
+    def convert_r_columns(c):
+        try:
+            first = c[0]
+            second = c[1:]
+            if first == 'X':
+                try:
+                    #  bingo! was X2015 R-style, return the integer
+                    return int(second)
+                except:
+                    # nope, not an int, fall down to final return statement
+                    pass
+        except:
+            # not a string/iterable/etc, fall down to final return statement
+            pass
+        return c
+    df.columns = df.columns.map(convert_r_columns)
+
     # if `value` is given but not `variable`,
     # melt value columns and use column name as `variable`
     if 'value' in kwargs and 'variable' not in kwargs:


Test patch:
diff --git a/tests/test_cast_to_iamc.py b/tests/test_cast_to_iamc.py
index c515cb4..c190b99 100644
--- a/tests/test_cast_to_iamc.py
+++ b/tests/test_cast_to_iamc.py
@@ -79,3 +79,21 @@ def test_cast_with_variable_and_value(meta_df):
 
     assert compare(pe_df, df).empty
     pd.testing.assert_frame_equal(df.data, pe_df.data.reset_index(drop=True))
+
+
+def test_cast_from_r_df(test_pd_df):
+    df = test_pd_df.copy()
+    # last two columns are years
+    df.columns = list(df.columns[:-2]) + ['X{}'.format(c)
+                                          for c in df.columns[-2:]]
+    obs = IamDataFrame(df)
+    exp = IamDataFrame(test_pd_df)
+    assert compare(obs, exp).empty
+    pd.testing.assert_frame_equal(obs.data, exp.data)
+
+
+def test_cast_from_r_df_err(test_pd_df):
+    df = test_pd_df.copy()
+    # last two columns are years
+    df.columns = list(df.columns[:-2]) + ['Xfoo', 'Xbar']
+    pytest.raises(ValueError, IamDataFrame, df)