Repository: calliope-project/calliope (language: python)

Issue:
Per-tech cyclic storage
## Problem description

Cyclic storage is currently either on or off model-wide. However, when some storage technologies are used as initial stock of a resource that should be depleted at the end of the model period, it would be useful to disable cyclic storage per-technology.




Discussion / hints:
Configuration should move into the default storage tech group, so it can be overridden globally or per tech.

Fix patch:
diff --git a/CHANGELOG.md b/CHANGELOG.md
index b31d11e4..ffa0b3e4 100644
--- a/CHANGELOG.md
+++ b/CHANGELOG.md
@@ -18,6 +18,8 @@ For instance, `flow_cap` can be defined per `carrier`.
 
 |new| Shadow prices obtained from a dual LP problem can be read by using `model.backend.shadow_prices.get("constraint_name")`
 
+|changed| |backwards-incompatible| `run.cyclic_storage` configuration option moved to being per-technology (`techs.tech_name.cyclic_storage: ...`).
+
 |changed| |backwards-incompatible| `file=/df=` parameter values as references to timeseries data is replaced with loading tabular data at the top-level using the `data_sources` key.
 
 |changed| Automatically derived transmission link distances default to kilometres, with the configuration option (`config.init.distance_unit`) to switch to the old default of distances in metres.
diff --git a/docs/advanced/constraints.md b/docs/advanced/constraints.md
index 81ce213c..89a36c59 100644
--- a/docs/advanced/constraints.md
+++ b/docs/advanced/constraints.md
@@ -170,12 +170,12 @@ In the latter case, Calliope will compute distances automatically based on the l
 
 For any technologies with storage (`storage` technologies or those with [storage buffer](#activating-storage-buffers-in-non-storage-technologies)), it is possible to link the storage at either end of the timeseries using the `cyclic_storage` parameter.
 This allows the user to better represent multiple years by just modelling one year.
-Cyclic storage is activated by default (to deactivate: `config.build.cyclic_storage: false`).
+Cyclic storage is activated by default (to deactivate: `cyclic_storage: false` in all storage technologies).
 As a result, a technology's initial stored energy at a given location will be equal to its stored energy at the end of the model's last timestep.
 
 For example, for a model running over a full year at hourly resolution, the initial storage at `Jan 1st 00:00` will be forced equal to the storage at the end of the timestep `Dec 31st 23:00`.
 By setting `storage_initial` for a technology, it is also possible to fix the value in the last timestep.
-For instance, with `config.build.cyclic_storage: true` and a `storage_initial` of zero, the stored energy *must* be zero by the end of the time horizon.
+For instance, with `cyclic_storage: true` and a `storage_initial: 0`, the stored energy *must* be zero by the end of the time horizon.
 
 Without cyclic storage in place, technologies can have any amount of stored energy by the end of the timeseries.
 This may prove useful in some cases, but makes little sense if you imagine your technologies operating in the same manner year-on-year.
diff --git a/docs/custom_math/syntax.md b/docs/custom_math/syntax.md
index 5588b75d..7697bb66 100644
--- a/docs/custom_math/syntax.md
+++ b/docs/custom_math/syntax.md
@@ -35,7 +35,7 @@ Configuration options are any that are defined in `config.build`, where you can
 
     ??? example annotate "Examples"
 
-        - If you want to apply a constraint only if the configuration option `config.build.cyclic_storage` is _True_, you would include `config.cyclic_storage=True` (`True`/`False` is case insensitive).
+        - If you want to apply a constraint only if the configuration option `config.build.mode` is _operate_, you would include `config.mode=operate`.
         - If you want to apply a constraint across all `nodes` and `techs`, but only where the `flow_eff` parameter is less than 0.5, you would include `flow_eff<0.5`.
         - If you want to apply a constraint only for the first timestep in your timeseries, you would include `timesteps=get_val_at_index(dim=timesteps, idx=0)`. (1)
         - If you want to apply a constraint only for the last timestep in your timeseries, you would include `timesteps=get_val_at_index(dim=timesteps, idx=-1)`.
@@ -67,7 +67,7 @@ These statements will be combined first.
 
 ??? example "Examples"
 
-    - If you want to apply a constraint for `storage` technologies if the configuration option `cyclic_storage` is activated and it is the last timestep of the series: `base_tech=storage and config.cyclic_storage=True and timesteps=get_val_at_index(dim=timesteps, idx=-1)`.
+    - If you want to apply a constraint for `storage` technologies if the configuration option `cyclic_storage` is activated and it is the last timestep of the series: `base_tech=storage and cyclic_storage=True and timesteps=get_val_at_index(dim=timesteps, idx=-1)`.
     - If you want to create a decision variable for the input carriers of conversion technologies: `carrier_in and base_tech=conversion`
     - If you want to apply a constraint if the parameter `source_unit` is `energy_per_area` or the parameter `area_use_per_flow_cap` is defined: `source_unit=energy_per_area or area_use_per_flow_cap`.
     - If you want to apply a constraint if the parameter `flow_out_eff` is less than or equal to 0.5 and `source_use` has been defined, or `flow_out_eff` is greater than 0.9 and `source_use` has not been defined: `(flow_out_eff<=0.5 and source_use) or (flow_out_eff>0.9 and not source_use)`.
diff --git a/docs/migrating.md b/docs/migrating.md
index 21511c3f..c5e2a183 100644
--- a/docs/migrating.md
+++ b/docs/migrating.md
@@ -515,6 +515,32 @@ Therefore, `24H` is equivalent to `24` in v0.6 if you are using hourly resolutio
 !!! warning
     Although we know that `operate` mode works on our example models, we have not introduced thorough tests for it yet - proceed with caution!
 
+### Per-technology cyclic storage
+
+The configuration option to set cyclic storage globally (`run.cyclic_storage`) has been moved to a parameter at the technology level.
+With this change, you can decide if a specific storage technology (or [technology with a storage buffer](#storage-buffers-in-all-technology-base-classes)) has cyclic storage enforced or not.
+As in v0.6, cyclic storage defaults to being _on_ (`cyclic_storage: true`).
+
+=== "v0.6"
+
+    ```yaml
+    run:
+      cyclic_storage: true
+    ```
+
+=== "v0.7"
+
+    ```yaml
+    techs:
+      storage_tech_with_cyclic_storage:
+        base_tech: storage
+        cyclic_storage: true
+      supply_tech_without_cyclic_storage:
+        base_tech: supply
+        include_storage: true
+        cyclic_storage: false
+    ```
+
 ## Removals
 
 ### `_equals` constraints
@@ -572,7 +598,7 @@ Instead, `supply_plus` can be effectively represented by using `supply` as the b
 To reimplement arbitrary links between carrier "tiers" (`in_2`, `out_2` etc.), you can define your own custom math, which is a simultaneously more powerful and more human-readable way of defining complex conversion technologies.
 
 !!! info "See also"
-    [Example of custom math to link carrier flows](examples/urban_scale/index.md#sparkles-interlude-custom-math).
+    [Example of custom math to link carrier flows](examples/urban_scale/index.md#interlude-custom-math).
 
 ### `carrier` key
 
@@ -670,8 +696,6 @@ Instead, data source filepaths should always be relative to the `model.yaml` fil
 Instead, it is possible to index any parameter over the time dimension.
 It is up to you to ensure the math formulation is set up to handle this change, which may require [tweaking existing math](custom_math/customise.md#introducing-custom-math-to-your-model).
 * With the [removal of time clustering](#clustering), we have removed `model.random_seed` and `model.time` options.
-* We have removed `model.objective`.
-With the introduction of our own math syntax, it is possible to add your own objective using [custom math](custom_math/index.md) and then activate/deactivate your objective of choice in the math.
 
 ### Plotting
 
diff --git a/src/calliope/config/config_schema.yaml b/src/calliope/config/config_schema.yaml
index 3251d0af..cccba5a2 100644
--- a/src/calliope/config/config_schema.yaml
+++ b/src/calliope/config/config_schema.yaml
@@ -83,12 +83,6 @@ properties:
             type: string
             default: pyomo
             description: Module with which to build the optimisation problem
-          cyclic_storage:
-            type: boolean
-            default: true
-            description: >
-              If true, link storage levels in the last model timestep with the first model timestep.
-              `inter_cluster_storage` custom math must be included if using time clustering and setting this to `true`.
           ensure_feasibility:
             type: boolean
             default: false
diff --git a/src/calliope/config/model_data_checks.yaml b/src/calliope/config/model_data_checks.yaml
index 6cf47b1d..051ebed6 100644
--- a/src/calliope/config/model_data_checks.yaml
+++ b/src/calliope/config/model_data_checks.yaml
@@ -38,4 +38,8 @@ fail:
   - where: (storage_cap_max OR storage_cap_min) AND storage_cap_per_unit
     message: Cannot define both `storage_cap_per_unit` and `storage_cap_max`/`storage_cap_min`
 
-warn: []
\ No newline at end of file
+warn:
+  - where: cyclic_storage=True AND (base_tech=storage OR include_storage=True) AND config.mode=operate
+    message: >-
+      Cyclic storage should be switched off in operate mode.
+      Since the parameter defaults to True, make sure you explicitly set `cyclic_storage: false` for all your technologies with storage.
diff --git a/src/calliope/config/model_def_schema.yaml b/src/calliope/config/model_def_schema.yaml
index 0ec41908..aee7b628 100644
--- a/src/calliope/config/model_def_schema.yaml
+++ b/src/calliope/config/model_def_schema.yaml
@@ -329,7 +329,7 @@ properties:
 
           integer_dispatch:
             type: boolean
-            default: False
+            default: false
             x-resample_method: first
             x-type: float
             title: Integer dispatch switch.
@@ -578,7 +578,7 @@ properties:
             title: Storage capacity per purchased unit.
             description: >-
               Set the storage capacity of each integer unit of a technology purchased.
-              Unit: Unit: $\frac{\text{energy}}{\text{unit}}$.
+              Unit: $\frac{\text{energy}}{\text{unit}}$.
 
           storage_discharge_depth:
             $ref: "#/$defs/TechParamNullNumber"
@@ -607,7 +607,18 @@ properties:
             title: Storage loss rate
             description: >-
                Rate of storage loss per hour, used to calculate lost stored flow as `(1 - storage_loss)^hours_per_timestep`.
-               Unit: Unit: $\frac{\text{fraction}}{\text{hour}}$.
+               Unit: $\frac{\text{fraction}}{\text{hour}}$.
+
+          cyclic_storage:
+            type: boolean
+            default: true
+            x-type: float
+            title: Cyclic storage switch.
+            description: >
+              If true, link storage levels in the last model timestep with the first model timestep.
+              `inter_cluster_storage` custom math must be included if using time clustering and setting this to `true`.
+              This must be set to `false` if using `operate` mode.
+              Unit: boolean.
 
           purchased_units_min_systemwide:
             $ref: "#/$defs/TechParamNullNumber"
diff --git a/src/calliope/example_models/national_scale/scenarios.yaml b/src/calliope/example_models/national_scale/scenarios.yaml
index d81b6cf6..6154e14c 100644
--- a/src/calliope/example_models/national_scale/scenarios.yaml
+++ b/src/calliope/example_models/national_scale/scenarios.yaml
@@ -72,7 +72,6 @@ overrides:
         mode: operate
         operate_window: 12H
         operate_horizon: 24H
-        cyclic_storage: false
     nodes:
       region1.techs.ccgt.flow_cap: 30000
 
@@ -94,6 +93,8 @@ overrides:
       region1_to_region1_1.flow_cap: 9000
       region1_to_region1_2.active: false
       region1_to_region1_3.flow_cap: 2281
+      csp.cyclic_storage: false
+      battery.cyclic_storage: false
 
   check_feasibility:
     config:
diff --git a/src/calliope/math/base.yaml b/src/calliope/math/base.yaml
index f8e19005..a34a216d 100644
--- a/src/calliope/math/base.yaml
+++ b/src/calliope/math/base.yaml
@@ -189,17 +189,17 @@ constraints:
       - expression: storage == $storage_previous_step + source_use * source_eff - flow_out_inc_eff
     sub_expressions:
       storage_previous_step: &storage_previous_step
-        - where: timesteps=get_val_at_index(timesteps=0) AND NOT config.cyclic_storage=True
+        - where: timesteps=get_val_at_index(timesteps=0) AND NOT cyclic_storage=True
           expression: storage_initial * storage_cap
         - where: >-
             (
-              (timesteps=get_val_at_index(timesteps=0) AND config.cyclic_storage=True)
+              (timesteps=get_val_at_index(timesteps=0) AND cyclic_storage=True)
               OR NOT timesteps=get_val_at_index(timesteps=0)
             ) AND NOT lookup_cluster_first_timestep=True
           expression: (1 - storage_loss) ** roll(timestep_resolution, timesteps=1) * roll(storage, timesteps=1)
         - where: >-
             lookup_cluster_first_timestep=True AND NOT
-            (timesteps=get_val_at_index(timesteps=0) AND NOT config.cyclic_storage=True)
+            (timesteps=get_val_at_index(timesteps=0) AND NOT cyclic_storage=True)
           expression: >-
             (1 - storage_loss) **
             select_from_lookup_arrays(timestep_resolution, timesteps=lookup_cluster_last_timestep) *
@@ -255,7 +255,7 @@ constraints:
       Fix the relationship between carrier stored in a `storage` technology at
       the start and end of the whole model period.
     foreach: [nodes, techs]
-    where: "storage AND storage_initial AND config.cyclic_storage=True"
+    where: "storage AND storage_initial AND cyclic_storage=True"
     equations:
       - expression: >-
           storage[timesteps=$final_step] * (
diff --git a/src/calliope/math/storage_inter_cluster.yaml b/src/calliope/math/storage_inter_cluster.yaml
index 95f96c45..ceb0ec0c 100644
--- a/src/calliope/math/storage_inter_cluster.yaml
+++ b/src/calliope/math/storage_inter_cluster.yaml
@@ -9,15 +9,15 @@ constraints:
   balance_supply_with_storage:
     sub_expressions:
       storage_previous_step: &storage_previous_step
-        - where: timesteps=get_val_at_index(timesteps=0) AND NOT config.cyclic_storage=True
+        - where: timesteps=get_val_at_index(timesteps=0) AND NOT cyclic_storage=True
           expression: storage_initial * storage_cap
         - where: >-
             (
-              (timesteps=get_val_at_index(timesteps=0) AND config.cyclic_storage=True)
+              (timesteps=get_val_at_index(timesteps=0) AND cyclic_storage=True)
               OR NOT timesteps=get_val_at_index(timesteps=0)
             ) AND NOT lookup_cluster_last_timestep
           expression: (1 - storage_loss) ** roll(timestep_resolution, timesteps=1) * roll(storage, timesteps=1)
-        - where: lookup_cluster_last_timestep AND NOT (timesteps=get_val_at_index(timesteps=0) AND NOT config.cyclic_storage=True)
+        - where: lookup_cluster_last_timestep AND NOT (timesteps=get_val_at_index(timesteps=0) AND NOT cyclic_storage=True)
           expression: "0"
 
   balance_storage:
@@ -83,14 +83,14 @@ constraints:
       - expression: storage_inter_cluster == $storage_previous_step + $storage_intra
     sub_expressions:
       storage_previous_step:
-        - where: datesteps=get_val_at_index(datesteps=0) AND NOT config.cyclic_storage=True
+        - where: datesteps=get_val_at_index(datesteps=0) AND NOT cyclic_storage=True
           expression: storage_initial
-        - where: (datesteps=get_val_at_index(datesteps=0) AND config.cyclic_storage=True) OR NOT datesteps=get_val_at_index(datesteps=0)
+        - where: (datesteps=get_val_at_index(datesteps=0) AND cyclic_storage=True) OR NOT datesteps=get_val_at_index(datesteps=0)
           expression: ((1 - storage_loss) ** 24) * roll(storage_inter_cluster, datesteps=1)
       storage_intra:
-        - where: datesteps=get_val_at_index(datesteps=0) AND NOT config.cyclic_storage=True
+        - where: datesteps=get_val_at_index(datesteps=0) AND NOT cyclic_storage=True
           expression: "0"
-        - where: NOT (datesteps=get_val_at_index(datesteps=0) AND NOT config.cyclic_storage=True)
+        - where: NOT (datesteps=get_val_at_index(datesteps=0) AND NOT cyclic_storage=True)
           expression: storage[timesteps=$final_step]
     slices:
       final_step:


Test patch:
diff --git a/tests/common/lp_files/balance_storage.lp b/tests/common/lp_files/balance_storage.lp
new file mode 100644
index 00000000..f7695498
--- /dev/null
+++ b/tests/common/lp_files/balance_storage.lp
@@ -0,0 +1,50 @@
+\* Source Pyomo model name=None *\
+
+min
+objectives(dummy_obj)(0):
++2.0 ONE_VAR_CONSTANT
+
+s.t.
+
+c_e_constraints(my_constraint)(b__test_storage__2005_01_01_00_00)_:
++1 variables(storage)(b__test_storage__2005_01_01_00_00)
++1.1111111111111112 variables(flow_out)(b__test_storage__electricity__2005_01_01_00_00)
+-1.0 variables(flow_in)(b__test_storage__electricity__2005_01_01_00_00)
+= 0.0
+
+c_e_constraints(my_constraint)(a__test_storage__2005_01_01_00_00)_:
++1 variables(storage)(a__test_storage__2005_01_01_00_00)
+-0.99 variables(storage)(a__test_storage__2005_01_01_01_00)
++1.1111111111111112 variables(flow_out)(a__test_storage__electricity__2005_01_01_00_00)
+-1.0 variables(flow_in)(a__test_storage__electricity__2005_01_01_00_00)
+= 0.0
+
+c_e_constraints(my_constraint)(a__test_storage__2005_01_01_01_00)_:
+-0.99 variables(storage)(a__test_storage__2005_01_01_00_00)
++1 variables(storage)(a__test_storage__2005_01_01_01_00)
++1.1111111111111112 variables(flow_out)(a__test_storage__electricity__2005_01_01_01_00)
+-1.0 variables(flow_in)(a__test_storage__electricity__2005_01_01_01_00)
+= 0.0
+
+c_e_constraints(my_constraint)(b__test_storage__2005_01_01_01_00)_:
+-0.99 variables(storage)(b__test_storage__2005_01_01_00_00)
++1 variables(storage)(b__test_storage__2005_01_01_01_00)
++1.1111111111111112 variables(flow_out)(b__test_storage__electricity__2005_01_01_01_00)
+-1.0 variables(flow_in)(b__test_storage__electricity__2005_01_01_01_00)
+= 0.0
+
+bounds
+   1 <= ONE_VAR_CONSTANT <= 1
+   0 <= variables(storage)(b__test_storage__2005_01_01_00_00) <= +inf
+   0 <= variables(flow_out)(b__test_storage__electricity__2005_01_01_00_00) <= +inf
+   0 <= variables(flow_in)(b__test_storage__electricity__2005_01_01_00_00) <= +inf
+   0 <= variables(storage)(a__test_storage__2005_01_01_00_00) <= +inf
+   0 <= variables(storage)(a__test_storage__2005_01_01_01_00) <= +inf
+   0 <= variables(flow_out)(a__test_storage__electricity__2005_01_01_00_00) <= +inf
+   0 <= variables(flow_in)(a__test_storage__electricity__2005_01_01_00_00) <= +inf
+   0 <= variables(flow_out)(a__test_storage__electricity__2005_01_01_01_00) <= +inf
+   0 <= variables(flow_in)(a__test_storage__electricity__2005_01_01_01_00) <= +inf
+   0 <= variables(storage)(b__test_storage__2005_01_01_01_00) <= +inf
+   0 <= variables(flow_out)(b__test_storage__electricity__2005_01_01_01_00) <= +inf
+   0 <= variables(flow_in)(b__test_storage__electricity__2005_01_01_01_00) <= +inf
+end
diff --git a/tests/common/test_model/energy_cap_per_storage_cap.yaml b/tests/common/test_model/energy_cap_per_storage_cap.yaml
index f737d456..5f34ea01 100644
--- a/tests/common/test_model/energy_cap_per_storage_cap.yaml
+++ b/tests/common/test_model/energy_cap_per_storage_cap.yaml
@@ -6,7 +6,6 @@ config:
 
   build:
     mode: plan
-    cyclic_storage: False # necessary so demand can be fed from stored energy
     ensure_feasibility: true
   solve:
     solver: cbc
@@ -23,6 +22,7 @@ techs:
     base_tech: storage
     storage_initial: 1.0
     lifetime: 60
+    cyclic_storage: false # necessary so demand can be fed from stored energy
     cost_flow_cap:
       data: 1500000
       index: monetary
diff --git a/tests/test_math.py b/tests/test_math.py
index 3f0b00c8..f14fa565 100644
--- a/tests/test_math.py
+++ b/tests/test_math.py
@@ -135,6 +135,21 @@ def test_balance_transmission(self, compare_lps):
         }
         compare_lps(model, custom_math, "balance_transmission")
 
+    def test_balance_storage(self, compare_lps):
+        "Test balance storage with one tech having and one tech not having per-tech cyclic storage."
+        self.TEST_REGISTER.add("constraints.balance_storage")
+        model = build_test_model(
+            {
+                "nodes.a.techs.test_storage.cyclic_storage": True,
+                "nodes.b.techs.test_storage.cyclic_storage": False,
+            },
+            "simple_storage,two_hours",
+        )
+        custom_math = {
+            "constraints": {"my_constraint": model.math.constraints.balance_storage}
+        }
+        compare_lps(model, custom_math, "balance_storage")
+
     @pytest.mark.xfail(reason="not all base math is in the test config dict yet")
     def test_all_math_registered(self, base_math):
         "After running all the previous tests in the class, the base_math dict should be empty, i.e. all math has been tested"