Repository: IAMconsortium/pyam
Pull request #292 (2019-11-15 12:26:15)

Log message nan data in init
# Please confirm that this PR has done the following:

- [x] Tests Added
- [x] Documentation Added (N/A)
- [x] Description in RELEASE_NOTES.md Added

## Adding to RELEASE_NOTES.md (remove section after adding to RELEASE_NOTES.md)

Please add a single line in the release notes similar to the following:

```
- (#XX)[http://link-to-pr.com] Added feature which does something
```

# Description of PR

A log message is now thrown if any null data is found on initialisation so that users know it's being dropped (I get caught out a lot being like, 'where has all my data gone'). Inspired by #291 and discussions in #286 


coveralls: 
[![Coverage Status](https://coveralls.io/builds/27013542/badge)](https://coveralls.io/builds/27013542)

Coverage decreased (-0.3%) to 85.067% when pulling **275803874abaae173de6a38d008a740d85779bc5 on znicholls:log-message-nan-data-in-init** into **103b60f8f2f46fddae4d0325cbc2651cb098030a on IAMconsortium:master**.

coveralls: 
[![Coverage Status](https://coveralls.io/builds/27013542/badge)](https://coveralls.io/builds/27013542)

Coverage decreased (-0.3%) to 85.067% when pulling **275803874abaae173de6a38d008a740d85779bc5 on znicholls:log-message-nan-data-in-init** into **103b60f8f2f46fddae4d0325cbc2651cb098030a on IAMconsortium:master**.

danielhuppmann: Thanks @znicholls for this PR. I can't test it right now because I don't see any logging output (see #293), but I am worried about confusion this may create for non-expert users.

We are aiming `pyam` (also) at a not-so-sophisticated user base, and with the proposed change, one would see a warning message (about something being dropped) if you load a snapshot (wide format) with some variables reported at a yearly resolution (e.g., temperature output from a stylized climate model) and other variables reported  at decadal resolution (e.g., IAM output). This could easily be misunderstood as the snapshot being corrupted or incomplete.
znicholls: > This could easily be misunderstood as the snapshot being corrupted or incomplete.

That's fair (but one could argue the snapshot is somehow incomplete as not everything is on the same timestep). Happy to also close this/take another route (only throw the warning if the dataframe is actually empty once you remove nan rows?).
danielhuppmann: > Only throw the warning if the dataframe is actually empty once you remove nan rows.

Writing to log already happens if an `IamDataFrame` is empty after `filter()`, see [here](https://github.com/IAMconsortium/pyam/blob/103b60f8f2f46fddae4d0325cbc2651cb098030a/pyam/core.py#L1001) - I think it makes sense to replicate the same in `__init__()`.

Maybe it makes sense to add an `empty` attribute similar to [pandas.DataFrame.empty](https://pandas.pydata.org/pandas-docs/stable/reference/api/pandas.DataFrame.empty.html)?
znicholls: > Writing to log already happens if an `IamDataFrame` is empty after `filter()`, see [here](https://github.com/IAMconsortium/pyam/blob/103b60f8f2f46fddae4d0325cbc2651cb098030a/pyam/core.py#L1001) - I think it makes sense to replicate the same in `__init__()`.

Nice will do that then
znicholls: > Maybe it makes sense to add an `empty` attribute similar to [pandas.DataFrame.empty](https://pandas.pydata.org/pandas-docs/stable/reference/api/pandas.DataFrame.empty.html)?

Smart

Patch:
diff --git a/RELEASE_NOTES.md b/RELEASE_NOTES.md
index faefb12..c868d77 100644
--- a/RELEASE_NOTES.md
+++ b/RELEASE_NOTES.md
@@ -1,6 +1,7 @@
 
 # Next Release
 
+- [#292](https://github.com/IAMconsortium/pyam/pull/292) Add warning message if `data` is empty at initialization (after formatting)
 - [#288](https://github.com/IAMconsortium/pyam/pull/288) Put `pyam` logger in its own namespace (see [here](https://docs.python-guide.org/writing/logging/#logging-in-a-library>))
 
 # Release v0.3.0
diff --git a/doc/source/data.rst b/doc/source/data.rst
index 5912e72..a396faf 100644
--- a/doc/source/data.rst
+++ b/doc/source/data.rst
@@ -1,15 +1,15 @@
 Data Model
 ==========
 
-Scenario data following the IAMC format
----------------------------------------
+The IAMC timeseries format for scenario data
+--------------------------------------------
 
 .. figure:: _static/iamc_logo.jpg
    :width: 120px
    :align: right
-   
+
    `IAMC website`_
-   
+
 .. _`IAMC Website`: http://www.globalchange.umd.edu/iamc/
 
 Over the past decade, the Integrated Assessment Modeling Consortium (IAMC)
@@ -68,10 +68,11 @@ In its original design, the IAMC data format (see above) assumed that the
 temporal dimension of any scenario data was restricted to full years
 represented as integer values.
 
-Two additional use cases are currently supported by :code:`pyam` in development
+Two additional use cases are currently supported by :class:`pyam` in development
 mode (beta):
 
- - using representative sub-annual timesteps
+ - using representative sub-annual timesteps via the :code:`extra_cols` feature
+   (see the section on `custom columns`_ in the :code:`data` table)
 
  - using continuous time via :class:`pandas.datetime`, replacing the name of
    the :code:`year` column by :code:`time`
@@ -79,42 +80,114 @@ mode (beta):
 Please reach out to the developers to get more information on this
 ongoing work.
 
-The :class:`pyam.IamDataFrame` class
-------------------------------------
+The :class:`IamDataFrame` class
+-------------------------------
 
 A :class:`pyam.IamDataFrame` instance is a wrapper for
-two :class:`pandas.DataFrame` instances (read the `docs`_):
-
- - :code:`data`: The data table is a dataframe containing the timeseries data
-   in "long format". It has the columns of the long data format :code:`['model',
-   'scenario', 'region', 'unit', 'year', 'value']`.
-
- - :code:`meta`: The meta table is a dataframe containing categorisation and
-   descriptive indicators. It has the index :code:`pyam.META_IDX = ['model',
-   'scenario']`.
-
-The standard output format is the IAMC-style "wide format", see the example
-above. This format can be accessed using :meth:`pyam.IamDataFrame.timeseries`,
-which returns a :class:`pandas.DataFrame` with the index :code:`pyam.IAMC_IDX =
-['model', 'scenario', 'region', 'variable', 'unit']` and the years as columns.
-
-.. _`docs`: https://pandas.pydata.org/pandas-docs/stable/reference/frame.html
+two :class:`pandas.DataFrame` instances (i.e., tables, see the `pandas docs`_
+for more information).
+
+.. _`pandas docs`: https://pandas.pydata.org/pandas-docs/stable/reference/frame.html
+
+The :code:`data` table
+~~~~~~~~~~~~~~~~~~~~~~
+
+This table contains the timeseries data related to an ensemble of scenarios.
+It is structured in *long format*, where each datapoint is one row. In contrast,
+the standard IAMC-style format is in *wide format* (see the example above),
+where each timeseries is one row and the timesteps are represented as columns.
+
+While long-format tables have advantages for the internal implementation of many
+:class:`pyam` functions, wide-format tables are more intuitive to users.
+The method :meth:`timeseries() <pyam.IamDataFrame.timeseries>` converts between
+the formats and returns a :class:`pandas.DataFrame` in wide format.
+Exporting an :class:`IamDataFrame` to file using
+:meth:`to_excel() <pyam.IamDataFrame.to_excel>` or
+:meth:`to_csv() <pyam.IamDataFrame.to_csv>` also writes the data table
+in wide format.
+
+The standard columns
+^^^^^^^^^^^^^^^^^^^^
+
+The columns of the :code:`data` table are :code:`['model', 'scenario', 'region',
+'unit', <time_format>, 'value']`, where :code:`time_format` is :code:`year`
+when timesteps are given in years (as :class:`int`) or :code:`time` when time
+is represented on a continuous scale (as :class:`pandas.datetime`).
+
+.. _`custom columns`:
+
+Custom columns of the :code:`data` table
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+
+If an :class:`IamDataFrame` is initialised with columns that are not in the
+list above nor interpreted as belonging to the time dimension (in wide format), these
+columns are included in the :code:`data` table as :code:`extra_cols`.
+This feature can be used, for example, to distinguish between multiple
+climate models providing different values for the variable
+:code:`Temperature|Global Mean`
+.
+
+.. warning::
+
+    Not all :class:`pyam` functions currently support the continuous
+    time or custom columns in a :code:`data` table. Please reach out via the 
+    mailing list or GitHub issues if you are not sure whether your use case
+    is supported.
+
+.. warning::
+
+    A word of warning for adding annotations relating to custom columns:
+    :class:`pyam` drops any data rows which have :code:`NaN` values.
+    Hence, if you are adding meta information to :code:`data`, you need to make
+    sure that you **add a value to every single row**.
+
+    The reason for that implementation is that pandas does not work as expected
+    with :code:`NaN` in many cases
+    (see `here <https://stackoverflow.com/a/18431417>`_ and
+    `here <https://stackoverflow.com/a/13606221>`_).
+    Therefore, it is simpler to remove :code:`NaN`'s to ensure that
+    :class:`pyam` has a clean dataset on which to operate.
+
+The :code:`meta` table
+~~~~~~~~~~~~~~~~~~~~~~
+
+This table is intended for categorisation and quantitative indicators at the
+model-scenario level. Examples in the `SR15`_ context are the warming category 
+('Below 1.5°C', '1.5°C with low overshoot', etc.) and the cumulative
+CO2 emissions until the end of the century.
+
+:class:`pyam` attempts to keep the information in :code:`meta` consistent with
+:code:`data` when performing operations (e.g.,
+:meth:`rename() <pyam.IamDataFrame.rename>`,
+:meth:`append() <pyam.IamDataFrame.append>`).
+See :meth:`utils.merge_meta() <pyam.utils.merge_meta>` for details.
+
+.. note::
+
+    The :code:`meta` table is not intended for annotations of individual
+    data points. If you want to add meta information at this level (e.g., 
+    which stylized climate model provided the variable
+    :code:`Temperature|Global Mean`, or whether a data point is from the 
+    original data source or the result of an operation), this should operate on
+    the :code:`data` table of the :class:`IamDataFrame` using the
+    custom-columns feature (see `custom columns`_ above).
 
 Filtering
 ---------
 
 The |pyam| package provides two methods for filtering scenario data:
 
-An existing `class`:IamDataFrame can be filtered using
-:meth:`pyam.IamDataFrame.filter(col=...) <pyam.IamDataFrame.filter>`,
-where :code:`col` can be any column of the
-:code:`data` table (i.e., `['model', 'scenario', 'region', 'unit', 'year']`)
-or any column of the :code:`meta` table. The returned object is
-a new :class:`pyam.IamDataFrame` instance.
-
-A :class:`pandas.DataFrame` with columns or index :code:`['model', 'scenario']`
-can be filtered by any :code:`meta` columns from a :code:`pyam.IamDataFrame`
-using :func:`pyam.filter_by_meta(data, df, col=..., join_meta=False) <pyam.filter_by_meta>`.
+An existing :class:`IamDataFrame` can be filtered using
+:meth:`filter(col=...) <pyam.IamDataFrame.filter>`,
+where :code:`col` can be any column of the :code:`data` table (i.e.,
+:code:`['model', 'scenario', 'region', 'unit', 'year'/'time']` or any `custom
+columns`_), or a column of the :code:`meta` table. The returned object is
+a new :class:`IamDataFrame` instance.
+
+A :class:`pandas.DataFrame` (:code:`data`) with columns or index
+:code:`['model', 'scenario']` can be filtered by any :code:`meta` columns from
+an :class:`IamDataFrame` (:code:`df`) using 
+:meth:`pyam.filter_by_meta(data, df, col=..., join_meta=False) <pyam.filter_by_meta>`.
 The returned object is a :class:`pandas.DataFrame` down-selected to those
 models-and-scenarios where the :code:`meta` column satisfies the criteria given
 by :code:`col=...` .
diff --git a/pyam/utils.py b/pyam/utils.py
index 75b9c18..58e8619 100644
--- a/pyam/utils.py
+++ b/pyam/utils.py
@@ -253,6 +253,12 @@ def format_data(df, **kwargs):
     if any(df[idx_cols].duplicated()):
         raise ValueError('duplicate rows in `data`!')
 
+    if df.empty:
+        logger.warning(
+            'Formatted data is empty! (perhaps there is a column full of '
+            'nans?)'
+        )
+
     return sort_data(df, idx_cols), time_col, extra_cols
 
 


Test patch:
diff --git a/tests/test_core.py b/tests/test_core.py
index 46e6b47..58fb1ef 100644
--- a/tests/test_core.py
+++ b/tests/test_core.py
@@ -1,4 +1,5 @@
 import os
+import logging
 import pytest
 import re
 import datetime
@@ -136,6 +137,21 @@ def test_init_datetime_subclass_long_timespan(test_pd_df):
     assert df["time"].min() == tmin
 
 
+def test_init_empty_message(test_pd_df, caplog):
+    tdf = test_pd_df.copy()
+    tdf["extra_col"] = "test_val"
+    tdf["extra_col"] = np.nan
+
+    res = IamDataFrame(tdf)
+    assert "Primary Energy|Coal" not in res.variables().tolist()
+
+    drop_message = (
+        "Formatted data is empty! (perhaps there is a column full of nans?)"
+    )
+    message_idx = caplog.messages.index(drop_message)
+    assert caplog.records[message_idx].levelno == logging.WARNING
+
+
 def test_to_excel(test_df):
     fname = 'foo_testing.xlsx'
     test_df.to_excel(fname)