Repository: e2nIEE/pandapower

Issue:
Time Series Analysis
Hello,
I´m currently working with time series in the IEEE 33-Bus Radial Distribution System and i have some implementation questions that i would like to clarifiy.

In the 33-bus network I had to add the time series to the loads and I noticed certain circumstances that support my doubts:

After adding the time series to the  loads, when I run the power flow are only recorded in the results (res_sgen, res_load, res_bus, among others...), the data related to the last row of the time series ( row 8759). On the other hand, when saving the results in excel (res_line.loading_percent, res_bus.vm_pu, among others...), through the OutputWriter, only the results related to row 0 and  row 8759 of the time series appear, instead of the 8760 rows.

This part of my code was executed the same way as in the youtube pandapower video (https://www.youtube.com/watch?v=sAHoJbfLhas) , regarding the time series. 
For example, in your case the results saved in excel should have the number of rows equivalent to the number of time steps, i.e. in the case of res_line.loading_percent you should have been able to see the loading percent for each time step (i.e. 95 line loadings for each line).  In my case, that I have time steps = (0,8759), I should be able to visualize the results for each time step (that is 8760 line loadings for each line), but in my excel file only the loading percent values of each line appear for time step= 0 and for time step= 8759 only, which I found a bit strange.

So, my first question refers to these facts previously mentioned. Shouldn't I be able to check the 8760 powerflows and save the results in excel of the 8760 time steps (no matter what results you want to save)?


My second question refers to the relationship between the active power assigned to loads (p_mw) and the values that are later passed from the time series. That is, initially a load has a value of p_mw defined, but when entering the time series the value of p_mw is updated (i.e. replaced directly) by the value of the time series. Should there not be a relationship between the installed power of the load (p_mw) and the time series value instead of direct replacement?  Because so, by this theory, the installed power (p_mw) defined before the input of the time series is as if it were irrelevant, given that the value that enters the power flow is the value of the time series...

Until this moment, these are the most critical doubts that I'd like to clarify, as well as to understand if what I executed in my script is correct!

Note: for the 33-bus network, I associated time series to the loads, being the structure of each one:

Time Series Consumption - (8760 rows x 32 columns), being the 32 columns, the columns to be passed to the 32 loads present in the network.

I send attached my 33-bus network script (.zip), as well as the excel file related to the time series (Consumption), for a better understanding of my specific doubts. 

[This_One_33.zip](https://github.com/e2nIEE/pandapower/files/4346712/This_One_33.zip)
[Consumption.xlsx](https://github.com/e2nIEE/pandapower/files/4346717/Consumption.xlsx)

I am very grateful for the attention provided and look forward to an answer in order to move on with my thesis project!

With gratitude,
Tiago Torres


Discussion:
As for my problem with the time series, I've already solved it. Just change time_steps = (0,8759) to time_steps=range(0,8759). To make sure that the problem was solved, I implemented the  time series into the loads through another similar way.

So, I ran two different methods (which you can check in my script, in the time series part ) based on these links :
- https://www.youtube.com/watch?v=sAHoJbfLhas 
- https://nbviewer.jupyter.org/github/e2nIEE/pandapower/blob/develop/tutorials/time_series.ipynb

I send attached my 33-bus network scripts(.zip) for a better understanding of my specific doubts.
[This_One_33.zip](https://github.com/e2nIEE/pandapower/files/4353018/This_One_33.zip)
[This_One_33_option.py.zip](https://github.com/e2nIEE/pandapower/files/4353019/This_One_33_option.py.zip)


However, despite having solved the previous problem, another even stranger problem has arisen...
After loading the results (res_bus.vm_pu, res_line.line_loading, res_load.p_mw,...) I noticed that from the "line" of excel 114 (index 112) all values (in each column) were repeated until the end. That is, up to the "row" of excel 114 all values seem to be correct (e.g.: in res_load.p_mw the values present in the dataframe of the time series - DATA_consumption, are checked), but after this "row" all values down (up to the "row" 8759) are repeated (repeats the value of "row" 114).

When I noticed this I thought that some must be wrong, because for example for the loads, the values that I should observe are the values present in the time series right?

Again, i am very grateful for the attention provided and look forward to an answer!



Hi Tiago, thanks for providing extensive examples for your problems. However, we don't have the capacity to look through a 260 line script to debug an error. If you want support, you will have to provide minimal working examples of your problem that are reproducable. Just google minimal working example, you will find a lot of examples and exaplanations about this... Can you please try to reduce your code into a MWE?
Hello again Leon,

Thank you in advance for your answer and I apologize for sending such extensive scripts. I realize that the way I sent my scripts is not easy to debug a specific error in the code, so this time I send my scripts as MWEs , that represent the ways I used to run the time series into the network.

In both of the scripts, the bug remains the same: repeated results (res_line, res_bus, res_load)  of the time step 112 until the last time step (8760).

In addition to the scripts I also attached the excel file that contains the values of the time series consumption ( for the loads).

[Consumption_MWE.xlsx](https://github.com/e2nIEE/pandapower/files/4361227/Consumption_MWE.xlsx)
[MWE.py.zip](https://github.com/e2nIEE/pandapower/files/4361230/MWE.py.zip)
[MWE_option.py.zip](https://github.com/e2nIEE/pandapower/files/4361234/MWE_option.py.zip)

Thanks again for your availability and help. I look forward to an answer!
Best regards,
Tiago


This is because `recycle=True` (which is default) in [ConstControl](https://pandapower.readthedocs.io/en/latest/control/controller.html#pandapower.control.controller.const_control.ConstControl) causes the powerflow to fail silently when diverging. In this case it diverges because the network is not able to deliver enough power to satisfy all of the loads for some timestep.

The values from the last converging timestep is reused for the rest of the timesteps, regardless of whether these converge or not. (which is why all the values become the same after the first divergence)

When explicitly setting `recycle=False`, the timeseries simulation will indeed crash and burn on the first failed time step, rather than failing silently. (You want divergence to cause a crash by default! Failing silently is bad.)

If you want the time series simulation to continue in the event of divergence (instead of crashing), you can set `continue_on_divergence=True` when calling [run_timeseries](https://pandapower.readthedocs.io/en/latest/timeseries/run_function.html#pandapower.timeseries.run_time_series.run_timeseries).

@lthurner @FlorianShepherd 
I think I've discovered the likely cause of this. Below is a minimal example to reproduce the underlying problem:

```py
import pandapower as pp
import pandapower.networks
net = pandapower.networks.case33bw()

# This succeeds:
try:
    pp.runpp(net, recycle=False)
except:
    print('ERROR: Powerflow diverged!')
print('Success:', net._ppc['success'])


# Now, the load is increased, causing divergence:
net.load.p_mw *= 10
try:
    pp.runpp(net, recycle=True)
except:
    print('ERROR: Powerflow diverged!')
print('Success:', net._ppc['success'])


# Try again, with recycle set to False
try:
    pp.runpp(net, recycle=False)
except:
    print('ERROR: Powerflow diverged!')
print('Success:', net._ppc['success'])
```

The resulting output is:
```
Success: True
ERROR: Powerflow diverged!
Success: True
ERROR: Powerflow diverged!
Success: False
```

The problem here is that the "success"-state is recycled to the next time step, and not changed when the network diverges.

Some potential ways of fixing this in the pandapower source code:
1. Set _ppc['success'] to false when the powerflow diverges.
2. Avoid copying over _ppc['success'] in the first place.

EDIT: Seems like the _ppc['et'] (elapsed time) and _ppc['iterations'] are also copied over, and unchanged upon divergence when recycle=True.
@bergkvist , I have to thank you in advance for your time and for clarifying my question.

However, I tried to execute the changes you recommended and the result was the same... From time step 112 ( last converging time step) all the values remain the same. 

When I called run_timeseries i set continue_on_divergence=True and there were no changes in the results. As well as I tried net._pcc['sucess'] = False and the result stayed the same.

That is, I'm only supposed to observe the correct results at each time step if there is convergence until time step 8759?

Attach here my minimal example as well as excel regarding the time series.  I'd appreciate if you check the necessary changes for my script to run as desired :)

[MWE.py.zip](https://github.com/e2nIEE/pandapower/files/4364754/MWE.py.zip)
[Consumption_MWE.xlsx](https://github.com/e2nIEE/pandapower/files/4364755/Consumption_MWE.xlsx)

Or I can show you:

```ruby
import pandapower as pp
import pandapower.networks as nw
import pandas as pd

from pandapower.control.controller.const_control import ConstControl
from pandapower.timeseries.data_sources.frame_data import DFData

from pandapower.timeseries import OutputWriter


#33 Bus Radial Distribution System
net = nw.case33bw()


#read the excel file from time series values (ton insert in all network loads)
# (8760 X 32)
df= pd.read_excel("/Users/tiagotorres/Desktop/Consumption_MWE.xlsx")

#drop this column (not needed)
df=df.drop(['Unnamed: 0'],axis=1)



#create the data source from it
dload=DFData(df)


#initialising ConstControl controller to update values of the loads ("load" elements)

ConstControl(net, "load", "p_mw", element_index= net.load.index, profile_name=df.columns, data_source= dload)

#initialising the outputwriter to save data to the current folder
ow= OutputWriter (net,time_steps = range(0,8759), output_path= "./results_1/", output_file_type= ".xlsx")


#adding the variables as specific columns i want, as outputs to be stored
ow.log_variable("res_load", "p_mw")


#starting the timeseries simulation for one year->  hourly values
#run_timeseries(net,time_steps = range(0,8759))


pp.timeseries.run_time_series.run_timeseries(net, time_steps=range(0,8759), continue_on_divergence=True)
```
Thanks again for your willingness to help!








@TiagoTorres1997 You also need to use `ConstControl(..., recycle=False)` when using `continue_on_divergence=True`. Otherwise it won't work, due to a problem with recycle.

For the time being, `recycle=True` is only safe to use if you know that every time step will converge. Note that `recycle=True` is default for ConstControl.
@TiagoTorres1997 
By the way, don't close this issue when you get it working. 

The problem with recycle=True is the deeper problem here - which you indirectly discovered. 

This is something that needs to be fixed in pandapower, and I think this issue should remain open as a reminder.
@bergkvist again, thank you very much for your help!
I will leave the issue open as requested.
With gratitude,
Tiago


Resolving patch:
diff --git a/pandapower/pf/run_newton_raphson_pf.py b/pandapower/pf/run_newton_raphson_pf.py
--- a/pandapower/pf/run_newton_raphson_pf.py
+++ b/pandapower/pf/run_newton_raphson_pf.py
@@ -72,7 +72,7 @@ def ppci_to_pfsoln(ppci, options):
 def _get_Y_bus(ppci, options, makeYbus, baseMVA, bus, branch):
     recycle = options["recycle"]
 
-    if recycle is not None and recycle is not False and not recycle["trafo"] and ppci["internal"]["Ybus"].size:
+    if isinstance(recycle, dict) and not recycle["trafo"] and ppci["internal"]["Ybus"].size:
         Ybus, Yf, Yt = ppci["internal"]['Ybus'], ppci["internal"]['Yf'], ppci["internal"]['Yt']
     else:
         # build admittance matrices
@@ -108,7 +108,7 @@ def _store_internal(ppci, internal_storage):
 
 def _get_Sbus(ppci, recycle=None):
     baseMVA, bus, gen, branch = ppci["baseMVA"], ppci["bus"], ppci["gen"], ppci["branch"]
-    if recycle is None or "Sbus" not in ppci["internal"]:
+    if not isinstance(recycle, dict) or "Sbus" not in ppci["internal"]:
         return makeSbus(baseMVA, bus, gen)
     if recycle["bus_pq"] or recycle["gen"]:
         return makeSbus(baseMVA, bus, gen)
diff --git a/pandapower/powerflow.py b/pandapower/powerflow.py
--- a/pandapower/powerflow.py
+++ b/pandapower/powerflow.py
@@ -100,6 +100,10 @@ def _recycled_powerflow(net, **kwargs):
 
     recycle = options["recycle"]
     ppc = net["_ppc"]
+    ppc["success"] = False
+    ppc["iterations"] = 0.
+    ppc["et"] = 0.
+
     if "bus_pq" in recycle and recycle["bus_pq"]:
         # update pq values in bus
         _calc_pq_elements_and_add_on_ppc(net, ppc)
@@ -123,6 +127,9 @@ def _recycled_powerflow(net, **kwargs):
 
     # run the Newton-Raphson power flow
     result = _run_newton_raphson_pf(ppci, options)
+    ppc["success"] = ppci["success"]
+    ppc["iterations"] = ppci["iterations"]
+    ppc["et"] = ppci["et"]
     if options["only_v_results"]:
         _ppci_bus_to_ppc(result, ppc)
         _ppci_other_to_ppc(result, ppc, options["mode"])
diff --git a/pandapower/run.py b/pandapower/run.py
--- a/pandapower/run.py
+++ b/pandapower/run.py
@@ -190,9 +190,9 @@ def runpp(net, algorithm='nr', calculate_voltage_angles="auto", init="auto",
         **recycle** (dict, none) - Reuse of internal powerflow variables for time series calculation
 
             Contains a dict with the following parameters:
-            _is_elements: If True in service elements are not filtered again and are taken from the last result in net["_is_elements"]
-            ppc: If True the ppc is taken from net["_ppc"] and gets updated instead of reconstructed entirely
-            Ybus: If True the admittance matrix (Ybus, Yf, Yt) is taken from ppc["internal"] and not reconstructed
+            bus_pq: If True PQ values of buses are updated
+            trafo: If True trafo relevant variables, e.g., the Ybus matrix, is recalculated
+            gen: If True Sbus and the gen table in the ppc are recalculated
 
         **neglect_open_switch_branches** (bool, False) - If True no auxiliary buses are created for branches when switches are opened at the branch. Instead branches are set out of service
 
@@ -201,7 +201,7 @@ def runpp(net, algorithm='nr', calculate_voltage_angles="auto", init="auto",
     # if dict 'user_pf_options' is present in net, these options overrule the net.__internal_options
     # except for parameters that are passed by user
     recycle = kwargs.get("recycle", None)
-    if (recycle is not None and recycle is not False) and _internal_stored(net):
+    if isinstance(recycle, dict) and _internal_stored(net):
         _recycled_powerflow(net, **kwargs)
         return
 
@@ -247,13 +247,6 @@ def rundcpp(net, trafo_model="t", trafo_loading="current", recycle=None, check_c
             - "current"- transformer loading is given as ratio of current flow and rated current of the transformer. This is the recommended setting, since thermal as well as magnetic effects in the transformer depend on the current.
             - "power" - transformer loading is given as ratio of apparent power flow to the rated apparent power of the transformer.
 
-        **recycle** (dict, none) - Reuse of internal powerflow variables for time series calculation
-
-            Contains a dict with the following parameters:
-            _is_elements: If True in service elements are not filtered again and are taken from the last result in net["_is_elements"]
-            ppc: If True the ppc (PYPOWER case file) is taken from net["_ppc"] and gets updated instead of reconstructed entirely
-            Ybus: If True the admittance matrix (Ybus, Yf, Yt) is taken from ppc["internal"] and not reconstructed
-
         **check_connectivity** (bool, False) - Perform an extra connectivity test after the conversion from pandapower to PYPOWER
 
             If true, an extra connectivity test based on SciPy Compressed Sparse Graph Routines is perfomed.


Test patch: