Repository: IAMconsortium/pyam
Pull request #101 (2018-10-01 06:29:55)

add a function to retrieve years in which a series crosses a threshold
# Please confirm that this PR has done the following:

- [x] Tests Added
- [x] Documentation Added
- [x] Description in RELEASE_NOTES.md Added

## Adding to RELEASE_NOTES.md (remove section after adding to RELEASE_NOTES.md)

Please add a single line in the release notes similar to the following:

```
- (#XX)[http://link-to-pr.com] Added feature which does something
```

# Description of PR

This PR adds a `pyam.cross_threshold()` function to retrieve a list of years in which a `pd.Series` (e.g., a timeseries row) crosses a given threshold.


Patch:
diff --git a/NOTICE.md b/NOTICE.md
index 545c35b..436599d 100644
--- a/NOTICE.md
+++ b/NOTICE.md
@@ -1,6 +1,6 @@
-   Copyright 2017 IIASA Energy Program
+   Copyright 2017-2018 IIASA Energy Program
 
-   The `pyam` package is licensed under 
+   The `pyam` package is licensed under
    the Apache License, Version 2.0 (the "License");
    you may not use the package except in compliance with the License.
    You may obtain a copy of the License at
diff --git a/README.md b/README.md
index 03585bb..0d74851 100644
--- a/README.md
+++ b/README.md
@@ -8,23 +8,23 @@ pyam: a Python toolkit for Integrated Assessment Modeling
 Overview and scope
 ------------------
 
-The ``pyam`` package provides a range of diagnostic tools and functions  
-for analyzing and working with IAMC-format timeseries data. 
+The ``pyam`` package provides a range of diagnostic tools and functions
+for analyzing and working with IAMC-format timeseries data.
 
 Features:
 - Summary of models, scenarios, variables, and regions included in a snapshot.
-- Display of timeseries data as pandas.DataFrame with IAMC-specific filtering 
+- Display of timeseries data as pandas.DataFrame with IAMC-specific filtering
   options.
 - Simple visualization and plotting functions.
-- Diagnostic checks for non-reported variables or timeseries data to identify 
+- Diagnostic checks for non-reported variables or timeseries data to identify
   outliers and potential reporting issues.
-- Categorization of scenarios according to timeseries data or meta-identifiers 
+- Categorization of scenarios according to timeseries data or meta-identifiers
   for further analysis.
 
-The package can be used with timeseries data that follows the data template 
+The package can be used with timeseries data that follows the data template
 convention of the [Integrated Assessment Modeling Consortium](http://www.globalchange.umd.edu/iamc/) (IAMC).
 An illustrative example is shown below;
-see [data.ene.iiasa.ac.at/database](http://data.ene.iiasa.ac.at/database/) 
+see [data.ene.iiasa.ac.at/database](http://data.ene.iiasa.ac.at/database/)
 for more information.
 
 | **model**    | **scenario** | **region** | **variable**   | **unit** | **2005** | **2010** | **2015** |
@@ -36,7 +36,7 @@ for more information.
 Tutorial
 --------
 
-A comprehensive tutorial for the basic functions is included 
+A comprehensive tutorial for the basic functions is included
 in [tutorial/pyam_first_steps](tutorial/pyam_first_steps.ipynb)
 using a partial snapshot of the IPCC AR5 scenario database.
 
@@ -49,16 +49,16 @@ See the instruction in [doc/README](doc/README.md).
 Authors
 -------
 
-This package was developed and is currently maintained 
-by Daniel Huppmann ([@danielhuppmann](https://github.com/danielhuppmann/))
-and Matthew Gidden ([@gidden](https://github.com/gidden)).
+This package was developed and is currently maintained
+by Matthew Gidden ([@gidden](https://github.com/gidden))
+and Daniel Huppmann ([@danielhuppmann](https://github.com/danielhuppmann/)).
 
 License
 -------
 
-Copyright 2017 IIASA Energy Program
+Copyright 2017-2018 IIASA Energy Program
 
-The ``pyam`` package is licensed 
+The ``pyam`` package is licensed
 under the Apache License, Version 2.0 (the "License");
 see [LICENSE](LICENSE) and [NOTICE](NOTICE.md) for details.
 
@@ -83,14 +83,14 @@ Installation instructions
 -------------------------
 
 0. Fork this repository and clone the forked repository (`<user>/pyam`)
-   to your machine. To fork the repository, look for the fork icon in the top 
+   to your machine. To fork the repository, look for the fork icon in the top
    right at [iiasa/pyam](https://github.com/iiasa/pyam).
    Add `iiasa/pyam` as `upstream` to your clone.
 
    *We recommend* [GitKraken](https://www.gitkraken.com/) *for users*
    *who prefer a graphical user interface application*
    *to work with Github (as opposed to the command line).*
-   
+
 ### Windows Users
 
 0. Double click on `install.bat` in the local folder where you cloned your fork.
diff --git a/RELEASE_NOTES.md b/RELEASE_NOTES.md
index 203bea5..035039a 100644
--- a/RELEASE_NOTES.md
+++ b/RELEASE_NOTES.md
@@ -1,6 +1,7 @@
 
 # Next Release
 
+- (#101)[https://github.com/IAMconsortium/pyam/pull/101] add function `cross_threshold()` to determine years where a timeseries crosses a given threshold
 - (#94)[https://github.com/IAMconsortium/pyam/pull/94] `set_meta()` can take pd.DataFrame (with columns `['model', 'scenario']`) as `index` arg
 - (#93)[https://github.com/IAMconsortium/pyam/pull/93] IamDataFrame can be initilialzed from pd.DataFrame with index
 - (#92)[https://github.com/IAMconsortium/pyam/pull/92] Adding `$` to the pseudo-regexp syntax in `pattern_match()`, adds override option
diff --git a/doc/source/conf.py b/doc/source/conf.py
index 42d8280..cad1a3e 100644
--- a/doc/source/conf.py
+++ b/doc/source/conf.py
@@ -73,8 +73,8 @@ master_doc = 'index'
 
 # General information about the project.
 project = u'pyam'
-copyright = u'2017-2018, Daniel Huppmann, Matthew Gidden @ IIASA Energy Program'
-author = u'Daniel Huppmann, Matthew Gidden'
+copyright = u'2017-2018, Matthew Gidden & Daniel Huppmann @ IIASA Energy Program'
+author = u'Matthew Gidden, Daniel Huppmann'
 
 # The version info for the project you're documenting, acts as replacement for
 # |version| and |release|, also used in various other places throughout the
diff --git a/doc/source/css/README.md b/doc/source/css/README.md
new file mode 100644
index 0000000..8b13789
--- /dev/null
+++ b/doc/source/css/README.md
@@ -0,0 +1,1 @@
+
diff --git a/doc/source/index.rst b/doc/source/index.rst
index e459d8c..c0815fe 100644
--- a/doc/source/index.rst
+++ b/doc/source/index.rst
@@ -1,11 +1,11 @@
-pyam: a Python toolkit for Integrated Assessment Modeling
-=========================================================
+**pyam**: an open-source Python package for IAM scenario analysis and visualization
+===================================================================================
 
 Overview and scope
 ------------------
 
-The ``pyam`` package provides a range of diagnostic tools and functions  
-for analyzing and working scenario data following the IAMC timeseries data format.
+The ``pyam`` package provides a range of diagnostic tools and functions
+for analyzing and visualising scenario data in the IAMC timeseries format.
 
 Features:
  - Summary of models, scenarios, variables, and regions included in a snapshot.
@@ -14,56 +14,99 @@ Features:
  - Advanced visualization and plotting functions.
  - Diagnostic checks for non-reported variables or timeseries values
    to analyze and validate scenario data.
- - Categorization of scenarios according to timeseries data 
-   or meta-identifiers for further analysis.
+ - Categorization of scenarios according to timeseries data
+   or metadata for further analysis.
 
-The package can be used with data that follows the data template convention 
+The package can be used with data that follows the data template convention
 of the `Integrated Assessment Modeling Consortium`_ (IAMC).
-An illustrative example is shown below; 
-see `data.ene.iiasa.ac.at/database`_ for more information.
+An illustrative example is shown below;
+see https://data.ene.iiasa.ac.at/database for more information.
 
-.. _`pandas.DataFrame`: 
+.. _`pandas.DataFrame`:
    https://pandas.pydata.org/pandas-docs/stable/generated/pandas.DataFrame.html
 
 .. _`Integrated Assessment Modeling Consortium`:
    http://www.globalchange.umd.edu/iamc/
 
-.. _`data.ene.iiasa.ac.at/database`: http://data.ene.iiasa.ac.at/database/
-
 ============  =============  ==========  ==============  ========  ========  ========  ========
-**model**     **scenario**   **region**  **variable**    **unit**  **2005**  **2010**  **2015**
+**Model**     **Scenario**   **Region**  **Variable**    **Unit**  **2005**  **2010**  **2015**
 ============  =============  ==========  ==============  ========  ========  ========  ========
-MESSAGE V.4   AMPERE3-Base   World       Primary Energy  EJ/y      454.5     479.6     ... 
+MESSAGE V.4   AMPERE3-Base   World       Primary Energy  EJ/y      454.5     479.6     ...
 ...           ...            ...         ...             ...       ...       ...       ...
 ============  =============  ==========  ==============  ========  ========  ========  ========
 
 
-License and source code repository
-----------------------------------
+License, source code, documentation
+-----------------------------------
 
 The `pyam` package is licensed under an `APACHE 2.0 open-source license`_.
 See the `LICENSE`_ file included in this repository for the full text.
-The source code is available on `github.com/IAMconsortium/pyam`_.
+
+The source code is available on https://github.com/IAMconsortium/pyam.
+The full documentation of the latest release is available on
+https://software.ene.iiasa.ac.at/pyam.
 
 .. _`APACHE 2.0 open-source license`: http://www.apache.org/licenses/LICENSE-2.0
 
 .. _`LICENSE`: https://github.com/IAMconsortium/pyam/blob/master/LICENSE
 
-.. _`github.com/IAMconsortium/pyam`: https://github.com/IAMconsortium/pyam
 
+The `pyam` data model
+---------------------
+
+Timeseries data
+^^^^^^^^^^^^^^^
+
+A `pyam.IamDataFrame` is a wrapper for two `pandas.DataFrame` instances:
+
+ - `data`: The data table is a dataframe containing the timeseries data in
+   "long format". It has the columns `pyam.LONG_IDX = ['model', 'scenario',
+   'region', 'unit', 'year', 'value']`.
+
+ - `meta`: The meta table is a dataframe containing categorisation and
+   descriptive indicators. It has the index `pyam.META_IDX = ['model',
+   'scenario']`.
+
+The standard output format is the IAMC-style "wide format", see the example
+above. This format can be accessed using `pd.IamDataFrame.timeseries()`,
+which returns a `pandas.DataFrame` with the index `pyam.IAMC_IDX = ['model',
+'scenario', 'region', 'variable', 'unit']` and the years as columns.
+
+Filtering
+^^^^^^^^^
+
+The `pyam` package provides two methods for filtering timeseries data:
+
+An existing IamDataFrame can be filtered using
+`pyam.IamDataFrame.filter(col=...)`_, where `col` can be any column of the
+`data` table (i.e., `['model', 'scenario', 'region', 'unit', 'year']) or any
+column of the `meta` table. The returned object is a new `pyam.IamDataFrame`
+instance.
+
+A `pandas.DataFrame` with columns or index `['model', 'scenario']` can be
+filtered by any `meta` columns from a `pyam.IamDataFrame` using
+`pyam.filter_by_meta(data, df, col=..., join_meta=False)`_. The returned
+object is a `pandas.DataFrame` downselected to those models-and-scenarios where
+the `meta` column satisfies the criteria given by `col=...` .
+Optionally, the `meta` columns are joined to the returned dataframe.
+
+ .. _`pyam.IamDataFrame.filter(col=...)` : IamDataFrame.html#pyam.IamDataFrame.filter
+
+ .. _`pyam.filter_by_meta(data, df, col=..., join_meta=False)` : pyam_functions.html
 
 `pyam` documentation
 --------------------
 
 See `this guide`_ for guidelines on NumPy/SciPy Documentation conventions.
 
-.. _`this guide`: 
+.. _`this guide`:
    https://github.com/numpy/numpy/blob/master/doc/HOWTO_DOCUMENT.rst.txt
 
-.. toctree:: 
-   :maxdepth: 2 
- 
+.. toctree::
+   :maxdepth: 2
+
    IamDataFrame
+   pyam_functions
    timeseries
    examples/index
 
diff --git a/doc/source/pyam_functions.rst b/doc/source/pyam_functions.rst
new file mode 100644
index 0000000..aa80e25
--- /dev/null
+++ b/doc/source/pyam_functions.rst
@@ -0,0 +1,6 @@
+.. currentmodule:: pyam
+
+Useful `pyam` functions
+=======================
+
+.. autofunction:: filter_by_meta
diff --git a/pyam/plotting.py b/pyam/plotting.py
index 9d15d56..c9b72fa 100644
--- a/pyam/plotting.py
+++ b/pyam/plotting.py
@@ -581,8 +581,9 @@ def _add_legend(ax, handles, labels, legend):
     if legend is None and len(labels) >= MAX_LEGEND_LABELS:
         logger().info('>={} labels, not applying legend'.format(
             MAX_LEGEND_LABELS))
-    legend = {} if legend in [True, None] else legend
-    ax.legend(handles, labels, **legend)
+    else:
+        legend = {} if legend in [True, None] else legend
+        ax.legend(handles, labels, **legend)
 
 
 def set_panel_label(label, ax=None, x=0.05, y=0.9):
diff --git a/pyam/timeseries.py b/pyam/timeseries.py
index 53e9c8a..f150f88 100644
--- a/pyam/timeseries.py
+++ b/pyam/timeseries.py
@@ -3,6 +3,9 @@
 import numpy as np
 import warnings
 
+from pyam.utils import (
+    isstr
+)
 
 # %%
 
@@ -82,3 +85,40 @@ def cumulative(x, first_year, last_year):
         value += x[last_year]
 
         return value
+
+
+def cross_threshold(x, threshold=0, direction=['from above', 'from below']):
+    """Returns a list of the years in which a timeseries (indexed over years)
+    crosses a given threshold
+
+    Parameters
+    ----------
+    x: pandas.Series
+        a timeseries indexed over years
+    threshold: float, default 0
+        the threshold that the timeseries is checked against
+    direction: str, optional, default `['from above', 'from below']`
+        whether to return all years where the threshold is crossed
+        or only where threshold is crossed in a specific direction
+    """
+    prev_yr, prev_val = None, None
+    years = []
+    direction = [direction] if isstr(direction) else list(direction)
+    if not set(direction).issubset(set(['from above', 'from below'])):
+        raise ValueError('invalid direction `{}`'.format(direction))
+
+    for yr, val in zip(x.index, x.values):
+        if np.isnan(val):  # ignore nans in the timeseries
+            continue
+        if prev_val is None:
+            prev_yr, prev_val = yr, val
+            continue
+        if not np.sign(prev_val - threshold) == np.sign(val - threshold):
+            if ('from above' in direction and prev_val > val) \
+                    or ('from below' in direction and prev_val < val):
+                change = (val - prev_val) / (yr - prev_yr)
+                # add one because int() rounds down
+                cross_yr = prev_yr + int((threshold - prev_val) / change) + 1
+                years.append(cross_yr)
+        prev_yr, prev_val = yr, val
+    return years


Test patch:
diff --git a/tests/test_timeseries.py b/tests/test_timeseries.py
new file mode 100644
index 0000000..ef60967
--- /dev/null
+++ b/tests/test_timeseries.py
@@ -0,0 +1,37 @@
+#!/usr/bin/env python3
+# -*- coding: utf-8 -*-
+
+import numpy as np
+import pandas as pd
+from pyam import cross_threshold
+import pytest
+
+
+def test_cross_treshold():
+    y = pd.Series(data=[np.nan, 1, 3, 1], index=[2002, 2005, 2007, 2013])
+    obs = cross_threshold(y, 2)
+    assert obs == [2007, 2011]
+
+
+def test_cross_treshold_empty():
+    y = pd.Series(data=[np.nan, 1, 3, 1], index=[2002, 2005, 2007, 2013])
+    obs = cross_threshold(y, 4)
+    assert obs == []
+
+
+def test_cross_treshold_from_below():
+    y = pd.Series(data=[np.nan, 1, 3, 1], index=[2002, 2005, 2007, 2013])
+    obs = cross_threshold(y, 2, direction='from below')
+    assert obs == [2007]
+
+
+def test_cross_treshold_from_above():
+    y = pd.Series(data=[np.nan, 1, 3, 1], index=[2002, 2005, 2007, 2013])
+    obs = cross_threshold(y, 2, direction='from above')
+    assert obs == [2011]
+
+
+def test_cross_treshold_direction_error():
+    y = pd.Series(data=[np.nan, 1, 3, 1], index=[2002, 2005, 2007, 2013])
+    pytest.raises(ValueError, cross_threshold, x=y, threshold=2,
+                  direction='up')