Repository: IAMconsortium/pyam
Pull request #709 (2022-10-17 09:59:52)

Ops nan hotfix
# Overview

As currently implemented, `fillna=0` is not caught by the line `if fillna:`, and is subsequently ignored. This fixes that to allow NaNs to be filled with 0s. Without this fix, scenario data is silently thrown away when doing computations.

I have confirmed that the newly added test fails on current main.

# Please confirm that this PR has done the following:

- [x] Tests Added
~- [ ] Documentation Added~
- [x] Name of contributors Added to AUTHORS.rst
- [x] Description in RELEASE_NOTES.md Added



codecov[bot]: # [Codecov](https://codecov.io/gh/IAMconsortium/pyam/pull/709?src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=IAMconsortium) Report
> Merging [#709](https://codecov.io/gh/IAMconsortium/pyam/pull/709?src=pr&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=IAMconsortium) (8733a4e) into [main](https://codecov.io/gh/IAMconsortium/pyam/commit/947d8c4df2dcb362baed6e1274d63a24142ba067?el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=IAMconsortium) (947d8c4) will **decrease** coverage by `0.0%`.
> The diff coverage is `90.0%`.

> :exclamation: Current head 8733a4e differs from pull request most recent head 068f63a. Consider uploading reports for the commit 068f63a to get more accurate results

```diff
@@           Coverage Diff           @@
##            main    #709     +/-   ##
=======================================
- Coverage   94.8%   94.8%   -0.1%     
=======================================
  Files         58      58             
  Lines       5899    5911     +12     
=======================================
+ Hits        5598    5608     +10     
- Misses       301     303      +2     
```


| [Impacted Files](https://codecov.io/gh/IAMconsortium/pyam/pull/709?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=IAMconsortium) | Coverage Δ | |
|---|---|---|
| [pyam/iiasa.py](https://codecov.io/gh/IAMconsortium/pyam/pull/709/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=IAMconsortium#diff-cHlhbS9paWFzYS5weQ==) | `86.7% <ø> (ø)` | |
| [tests/test\_iiasa.py](https://codecov.io/gh/IAMconsortium/pyam/pull/709/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=IAMconsortium#diff-dGVzdHMvdGVzdF9paWFzYS5weQ==) | `97.8% <ø> (-0.1%)` | :arrow_down: |
| [tests/test\_io.py](https://codecov.io/gh/IAMconsortium/pyam/pull/709/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=IAMconsortium#diff-dGVzdHMvdGVzdF9pby5weQ==) | `98.4% <85.7%> (-1.6%)` | :arrow_down: |
| [pyam/core.py](https://codecov.io/gh/IAMconsortium/pyam/pull/709/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=IAMconsortium#diff-cHlhbS9jb3JlLnB5) | `94.5% <87.5%> (ø)` | |
| [pyam/\_ops.py](https://codecov.io/gh/IAMconsortium/pyam/pull/709/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=IAMconsortium#diff-cHlhbS9fb3BzLnB5) | `94.0% <100.0%> (ø)` | |
| [pyam/utils.py](https://codecov.io/gh/IAMconsortium/pyam/pull/709/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=IAMconsortium#diff-cHlhbS91dGlscy5weQ==) | `91.6% <100.0%> (-0.1%)` | :arrow_down: |
| [tests/test\_ops.py](https://codecov.io/gh/IAMconsortium/pyam/pull/709/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=IAMconsortium#diff-dGVzdHMvdGVzdF9vcHMucHk=) | `100.0% <100.0%> (ø)` | |

Help us with your feedback. Take ten seconds to tell us [how you rate us](https://about.codecov.io/nps?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=IAMconsortium). Have a feature suggestion? [Share it here.](https://app.codecov.io/gh/feedback/?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=IAMconsortium)


Patch:
diff --git a/RELEASE_NOTES.md b/RELEASE_NOTES.md
index 32fb199..5db0c6d 100644
--- a/RELEASE_NOTES.md
+++ b/RELEASE_NOTES.md
@@ -7,6 +7,7 @@ Bump minimum version of **pandas** to v1.2.0 to support automatic engine selecti
 
 ## Individual updates
 
+- [#709](https://github.com/IAMconsortium/pyam/pull/709) Hotfix ops to support `fillna=0`
 - [#708](https://github.com/IAMconsortium/pyam/pull/708) Remove 'xls' as by-default-supported file format
 
 # Release v1.6.0
diff --git a/pyam/_ops.py b/pyam/_ops.py
index ca1477a..5160795 100644
--- a/pyam/_ops.py
+++ b/pyam/_ops.py
@@ -104,7 +104,7 @@ def _op_data(df, name, method, axis, fillna=None, args=(), ignore_units=False, *
             )
 
     # merge all args and kwds that are based on `df._data` to apply fillna
-    if fillna:
+    if fillna is not None:
         _data_cols = [_args[i] for i, is_data in enumerate(_data_args) if is_data]
         _data_cols += [kwds[key] for key, is_data in _data_kwds.items() if is_data]
         _data = pd.merge(*_data_cols, how="outer", left_index=True, right_index=True)


Test patch:
diff --git a/tests/test_ops.py b/tests/test_ops.py
index 149a2dd..3ef764c 100644
--- a/tests/test_ops.py
+++ b/tests/test_ops.py
@@ -38,6 +38,15 @@ def df_ops_variable_number(func, variable, unit, meta):
     return IamDataFrame(_data.T, **DF_ARGS, variable=variable, unit=unit, meta=meta)
 
 
+def df_ops_fillna_0(func, variable, unit, meta):
+    """Return IamDataFrame when performing operation on test_df with default (5)"""
+    _data = pd.DataFrame(
+        [["scen_a", "scen_b"], [func(1, 0.5), func(2, 0)], [func(6, 3), func(7, 0)]],
+        index=DF_INDEX,
+    )
+    return IamDataFrame(_data.T, **DF_ARGS, variable=variable, unit=unit, meta=meta)
+
+
 def test_add_raises(test_df_year):
     """Calling an operation with args that don't return an IamDataFrame raises"""
     match = "Value returned by `add` cannot be cast to an IamDataFrame: 5"
@@ -49,6 +58,7 @@ def test_add_raises(test_df_year):
     "arg, df_func, fillna, ignore_units",
     (
         ("Primary Energy|Coal", df_ops_variable, None, False),
+        ("Primary Energy|Coal", df_ops_fillna_0, 0, "foo"),
         ("Primary Energy|Coal", df_ops_variable_default, {"c": 7, "b": 5}, "foo"),
         ("Primary Energy|Coal", df_ops_variable_default, 5, "foo"),
         (registry.Quantity(2, "EJ/yr"), df_ops_variable_number, None, False),
@@ -81,8 +91,8 @@ def test_add_variable(test_df_year, arg, df_func, fillna, ignore_units, append):
         if ignore_units:
             with pytest.raises(pint.UndefinedUnitError):
                 test_df_year.add(*args, fillna=fillna, ignore_units=False)
-
-        assert_iamframe_equal(exp, test_df_year.add(*args, **kwds))
+        obs = test_df_year.add(*args, **kwds)
+        assert_iamframe_equal(exp, obs)
 
 
 @pytest.mark.parametrize("append", (False, True))