Repository: e2nIEE/pandapower

Issue:
Large indices lead to high memory consumption
Creating a network with large indices e.g. when creating generators leads to high memory consumption.

The problem is the line `lookup = -np.ones(max(pandapower_index) + 1, dtype=int)` when building the lookup:

https://github.com/lthurner/pandapower/blob/aef97f1f16ab496dd7d0b772c486b6e3be566b93/pandapower/pd2ppc.py#L255-L269

If one index is very large this creates a huge (mostly empty) ndarray, filling the memory. While I have not checked with the rest of the code this seemes unnecessary.

The same problem exists when building the bus lookup:
https://github.com/lthurner/pandapower/blob/aef97f1f16ab496dd7d0b772c486b6e3be566b93/pandapower/build_bus.py#L109-L117

For buses a workaround is to call `pandapower.create_continuous_bus_index(net, start=0)` before running the powerflow.

The problem only becomes significant for large indices of greater than 100000000 which might be unusual but occured in some of my calculations. For smaller indices it still seems like a waste of memory though.



Discussion:
Our tests showed that the current implementation of the bus lookup (+numba) is the most performant one. So we decided to trade a slightly higher memory usage with increased performance (the resulting memory footprint is negligible for reasonable maximal bus indizes). 

In order to let unaware user know about this peculiarity we log a warning in case large bus indize values are encountered.

I was not aware that we use a similar implementation for element lookup. Maybe we should log a warning here too.

The problem should be easily solved by net.gen.reset_index(inplace=True) ?

Thanks for the explanation.
 If you have done the tests I am the last person who stands in the way of a performance increase by wasting a bit of memory, which in most cases should indeed be insignificant. However a warning would be nice.

But I wonder if you have considered very sparse indices in your performance tests. I would suspect that the performance/number of elements would decrease with a more sparse indices?

The data I am dealing with at the moment uses one continuus index for all types of network elements, which leads to a sparse index for one type of element.

Initally I used `create_continuous_bus_index` and ignored the index for all other elements to make it run at all, but this makes comparison with the original data very hard.

Luckily I can process the original data for a more compact index, since already for that data an index of > 100000000 is not necessary.

So I still would recommend a warning and I still might hit this problem in the future when dealing with larger data sets...


FYI, a little bit of history: The first implementation was with dicts and very low memory consumption, with 
key = pandapower index, val = ppc index

This was really slow for large arrays since it was necessary to iterate over every key in the dict. Since every index is unique we switched to the array where 
array_index = pandapower index
array_val = ppc_index

As ascheidl said, this is the fastest solution we found, but has problems with large indices.

Maybe a sparse lookup would be a compromise here. Thank you for the hint!
One additional note on using very high index numbers: a use case where this happens if you have some identifying number that you want to use as an index. I don't know if that's your usecase or not, but if so, an alternative to using the id as an index is to just use continuous indices when creating the net, and put the IDs in a new column, e.g. net.bus.ID. This way you avoid the memory problem, and you also avoid losing leading or trailing zeros. Because if you have an ID such as 0125810, this would become index 12581 and you might have trouble identifying it later on. If you save it in a new column as a string, all information can be preserved.


Regarding the idea of a sparse lookup array, I tested it with a pandas SparseArray. The access times are significantly higher with the sparse solution, especially if not one value is accessed, but many at one time:

>dense = np.random.randn(1000000)
sparse = pd.SparseArray(dense)

>%timeit dense[500:1000]
631 ns ± 36.9 ns per loop (mean ± std. dev. of 7 runs, 1000000 loops each)

>%timeit sparse[500:1000]
55.9 ms ± 7.75 ms per loop (mean ± std. dev. of 7 runs, 10 loops each)

The sparse array is more efficient in memory, but the access time for multiple values is almost constant for dense arrays. It does not scale well for the sparse version, because premusably values are accessed in a loop. So I don't think thats a viable alternative.

#edit: The first implementation with dictionaries in comparison:
>lookup = {i: i for i in range(500, 1000)}

>%timeit [lookup[i] for i in range(500, 1000)]
115 µs ± 6.92 µs per loop (mean ± std. dev. of 7 runs, 10000 loops each)

Resolving patch:
diff --git a/pandapower/auxiliary.py b/pandapower/auxiliary.py
--- a/pandapower/auxiliary.py
+++ b/pandapower/auxiliary.py
@@ -399,6 +399,17 @@ def _check_bus_index_and_print_warning_if_high(net, n_max=1e7):
             " Try resetting the bus indices with the toolbox function "
             "create_continous_bus_index()" % max_bus)
 
+def _check_gen_index_and_print_warning_if_high(net, n_max=1e7):
+    if net.gen.empty:
+        return
+    max_gen = max(net.gen.index.values)
+    if max_gen >= n_max and len(net["gen"]) < n_max:
+        logger.warning(
+            "Maximum generator index is high (%i). You should avoid high generator indices because of perfomance reasons."
+            #" Try resetting the bus indices with the toolbox function "
+            #"create_continous_bus_index()"
+            % max_gen)
+
 
 def _add_pf_options(net, tolerance_kva, trafo_loading, numba, ac,
                     algorithm, max_iteration, **kwargs):
diff --git a/pandapower/run.py b/pandapower/run.py
--- a/pandapower/run.py
+++ b/pandapower/run.py
@@ -7,7 +7,8 @@
 import numpy as np
 
 from pandapower.auxiliary import _add_pf_options, _add_ppc_options, _add_opf_options, \
-    _check_if_numba_is_installed, _check_bus_index_and_print_warning_if_high
+    _check_if_numba_is_installed, _check_bus_index_and_print_warning_if_high, \
+    _check_gen_index_and_print_warning_if_high
 from pandapower.optimal_powerflow import _optimal_powerflow
 from pandapower.opf.validate_opf_input import _check_necessary_opf_parameters
 from pandapower.powerflow import _powerflow
@@ -244,6 +245,7 @@ def runpp(net, algorithm='nr', calculate_voltage_angles="auto", init="auto", max
     # net.__internal_options.update(overrule_options)
     net._options.update(overrule_options)
     _check_bus_index_and_print_warning_if_high(net)
+    _check_gen_index_and_print_warning_if_high(net)
     _powerflow(net, **kwargs)
 
 
@@ -310,6 +312,7 @@ def rundcpp(net, trafo_model="t", trafo_loading="current", recycle=None, check_c
     _add_pf_options(net, tolerance_kva=tolerance_kva, trafo_loading=trafo_loading,
                     numba=numba, ac=ac, algorithm=algorithm, max_iteration=max_iteration)
     _check_bus_index_and_print_warning_if_high(net)
+    _check_gen_index_and_print_warning_if_high(net)
     _powerflow(net, **kwargs)
 
 
@@ -385,6 +388,7 @@ def runopp(net, verbose=False, calculate_voltage_angles=False, check_connectivit
                      voltage_depend_loads=False, delta=delta)
     _add_opf_options(net, trafo_loading=trafo_loading, ac=ac, numba=numba)
     _check_bus_index_and_print_warning_if_high(net)
+    _check_gen_index_and_print_warning_if_high(net)
     _optimal_powerflow(net, verbose, suppress_warnings, **kwargs)
 
 
@@ -446,4 +450,5 @@ def rundcopp(net, verbose=False, check_connectivity=True, suppress_warnings=True
                      voltage_depend_loads=False, delta=delta)
     _add_opf_options(net, trafo_loading=trafo_loading, ac=ac)
     _check_bus_index_and_print_warning_if_high(net)
+    _check_gen_index_and_print_warning_if_high(net)
     _optimal_powerflow(net, verbose, suppress_warnings, **kwargs)


Test patch: