Repository: IAMconsortium/pyam
Pull request #579 (2021-09-10 05:58:54)

Improve performance of IamDataFrame initialization
# Please confirm that this PR has done the following:

- ~Tests Added~ -> no functional changes, existing tests should cover all cases where something could go wrong
- [x] Name of contributors Added to AUTHORS.rst
- [x] Description in RELEASE_NOTES.md Added

# Description of PR

This PR implement faster and less memory-intensive initialization of an IamDataframe by doing the sorting and validation on the indexed pd.Series instead of the DataFrame. Based on memory-profiling work by @phackstock.


codecov[bot]: # [Codecov](https://codecov.io/gh/IAMconsortium/pyam/pull/579?src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=IAMconsortium) Report
> Merging [#579](https://codecov.io/gh/IAMconsortium/pyam/pull/579?src=pr&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=IAMconsortium) (8a29b5c) into [main](https://codecov.io/gh/IAMconsortium/pyam/commit/5f10925022f90f2b6b2ecbef14e935b38e3852a2?el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=IAMconsortium) (5f10925) will **decrease** coverage by `0.1%`.
> The diff coverage is `100.0%`.

[![Impacted file tree graph](https://codecov.io/gh/IAMconsortium/pyam/pull/579/graphs/tree.svg?width=650&height=150&src=pr&token=ThxWlNkonJ&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=IAMconsortium)](https://codecov.io/gh/IAMconsortium/pyam/pull/579?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=IAMconsortium)

```diff
@@           Coverage Diff           @@
##            main    #579     +/-   ##
=======================================
- Coverage   93.7%   93.5%   -0.2%     
=======================================
  Files         50      50             
  Lines       5336    5336             
=======================================
- Hits        5001    4993      -8     
- Misses       335     343      +8     
```


| [Impacted Files](https://codecov.io/gh/IAMconsortium/pyam/pull/579?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=IAMconsortium) | Coverage Δ | |
|---|---|---|
| [pyam/utils.py](https://codecov.io/gh/IAMconsortium/pyam/pull/579/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=IAMconsortium#diff-cHlhbS91dGlscy5weQ==) | `91.6% <100.0%> (ø)` | |
| [pyam/datareader.py](https://codecov.io/gh/IAMconsortium/pyam/pull/579/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=IAMconsortium#diff-cHlhbS9kYXRhcmVhZGVyLnB5) | `50.0% <0.0%> (-50.0%)` | :arrow_down: |
| [tests/test\_datareader.py](https://codecov.io/gh/IAMconsortium/pyam/pull/579/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=IAMconsortium#diff-dGVzdHMvdGVzdF9kYXRhcmVhZGVyLnB5) | `69.5% <0.0%> (-13.1%)` | :arrow_down: |

------

[Continue to review full report at Codecov](https://codecov.io/gh/IAMconsortium/pyam/pull/579?src=pr&el=continue&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=IAMconsortium).
> **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=IAMconsortium)
> `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
> Powered by [Codecov](https://codecov.io/gh/IAMconsortium/pyam/pull/579?src=pr&el=footer&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=IAMconsortium). Last update [5f10925...8a29b5c](https://codecov.io/gh/IAMconsortium/pyam/pull/579?src=pr&el=lastupdated&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=IAMconsortium). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=IAMconsortium).

danielhuppmann: Thanks to @sandrinecharousset who reported an "out-of-memory" issue when loading large files as IamDataFrame - which motivated @phackstock and me to dig a bit into the performance...
danielhuppmann: @gidden @znicholls @Rlamboll @coroa, any thoughts? If there are no objections, I'll merge next week.
Rlamboll: Looks good to me and it's always good to speed things up a little! 
znicholls: Right I've now read the whole thing so have answered my own question, all good
danielhuppmann: Thanks @phackstock for the effort, thanks @znicholls & @Rlamboll for taking a look at the implementation!

Patch:
diff --git a/AUTHORS.rst b/AUTHORS.rst
index a9c863a..a5ab0bb 100644
--- a/AUTHORS.rst
+++ b/AUTHORS.rst
@@ -20,3 +20,4 @@ The following persons contributed to the development of the |pyam| framework:
 - Patrick Jürgens `@pjuergens <https://github.com/pjuergens>`_
 - Florian Maczek `@macflo8 <https://github.com/macflo8>`_
 - Laura Wienpahl `@LauWien <https://github.com/LauWien>`_
+- Philip Hackstock `@phackstock <https://github.com/phackstock>`_
diff --git a/RELEASE_NOTES.md b/RELEASE_NOTES.md
index 5b99b9c..d125589 100644
--- a/RELEASE_NOTES.md
+++ b/RELEASE_NOTES.md
@@ -6,6 +6,7 @@ The dependencies were updated to require `xlrd>=2.0` (previously `<2.0`) and `op
 
 ## Individual updates
 
+- [#579](https://github.com/IAMconsortium/pyam/pull/579) Increase performance of IamDataFrame initialization
 - [#572](https://github.com/IAMconsortium/pyam/pull/572) Unpinned the requirements for xlrd and added openpyxl as a requirement to ensure ongoing support of both `.xlsx` and `.xls` files out of the box
 
 # Release v1.1.0
diff --git a/pyam/utils.py b/pyam/utils.py
index 1fb7520..f486dee 100644
--- a/pyam/utils.py
+++ b/pyam/utils.py
@@ -329,21 +329,28 @@ def format_data(df, index, **kwargs):
     df.loc[df.unit.isnull(), "unit"] = ""
 
     # verify that there are no nan's left (in columns)
-    null_rows = df.isnull().values
+    null_rows = df.isnull().T.any()
     if null_rows.any():
-        _raise_data_error("empty cells in `data`", df.loc[null_rows])
+        _raise_data_error("Empty cells in `data`", df.loc[null_rows])
+    del null_rows
 
-    # check for duplicates and empty data
+    # format the time-column
+    df = format_time_col(df, time_col)
+
+    # cast to pd.Series, check for duplicates
     idx_cols = index + REQUIRED_COLS + [time_col] + extra_cols
-    rows = df[idx_cols].duplicated()
-    if any(rows):
-        _raise_data_error("duplicate rows in `data`", df.loc[rows, idx_cols])
+    df = df.set_index(idx_cols).value
 
+    rows = df.index.duplicated()
+    if any(rows):
+        _raise_data_error(
+            "Duplicate rows in `data`", df[rows].index.to_frame(index=False)
+        )
+    del rows
     if df.empty:
         logger.warning("Formatted data is empty!")
 
-    df = format_time_col(sort_data(df, idx_cols), time_col)
-    return df.set_index(idx_cols).value, index, time_col, extra_cols
+    return df.sort_index(), index, time_col, extra_cols
 
 
 def format_time_col(data, time_col):


Test patch:
diff --git a/tests/test_core.py b/tests/test_core.py
index e4501ea..a8c2988 100644
--- a/tests/test_core.py
+++ b/tests/test_core.py
@@ -80,7 +80,7 @@ def test_init_df_with_float_cols_raises(test_pd_df):
 def test_init_df_with_duplicates_raises(test_df):
     _df = test_df.timeseries()
     _df = _df.append(_df.iloc[0]).reset_index()
-    match = "3  model_a   scen_a  World  Primary Energy  EJ/yr"
+    match = "0  model_a   scen_a  World  Primary Energy  EJ/yr"
     with pytest.raises(ValueError, match=match):
         IamDataFrame(_df)