Repository: PowerGridModel/power-grid-model (language: cpp)

Issue:
[FEATURE] Add support columnar data buffer to save memory usage
# Background

Power Grid Model currently uses the row-based buffer to share the data across the C-API. For example, the memory layout of a node input buffer looks like:

```
| XXXX OOOO  XXXXXXXX  | XXXX OOOO  XXXXXXXX  | ... |
| id_0       u_rated_0 | id_1       u_rated_1 | ... |
```

Where `X` is a meaningful byte and `O` is an empty byte to align the memory.

In this way we can match the input/update/output data structs exactly as we do in the calculation core. This can deliver the performance benefits as we avoid any copies and the memory layout is exactly matching.

When we need to leave some attributes unspecified in input/update, we set the pre-defined [`null`](https://power-grid-model.readthedocs.io/en/stable/advanced_documentation/native-data-interface.html#basic-data-types) value defined in the place so the core knows the value should be ignored.

# Problem

While this design is CPU-wise very efficient, it could be memory-wise inefficient due to several reasons:

1. If you have many unspecified attributes, you waste the memory to have many `null`.
2. To align the data structure, the compiler will add paddings, as shown above. Those memory will not be used at all.
3. If we are only interested in certain attributes in the output, we still need to create the memory buffer for all the attributes. See the related issue here: #542.

There is a strong case to support columnar data buffers. We give two real-world examples of this issue.

## Example of update buffer

If we have an update buffer of 1000 scenarios of 1000 `sym_load`, the buffer size is `24 * 1000 * 1000 = 24,000,000` bytes. However, we might only need to specify `id` and `p_specified`. If we could provide these two array separately, the buffer size in total is `(8 + 4) * 1000 * 1000 = 12,000,000` bytes. The reduction on memory footprint is 50%!

## Example of output buffer

If we get a result buffer of 1000 scenarios of 1000 `line`, the buffer size is `80 * 1000 * 1000 = 80,000,000` bytes. However, we might only need to know the `loading` output, not even `id`, since we already know the `id` order in the input. The buffer size is `8 * 1000 * 1000 = 8,000,000` bytes. We can save 90% of memory footprint!

# Proposal

We propose to support columnar data buffers across the C-API (and further in the Python API). Both the PGM core and serialization need to support that.

## C-API

We already have the `dataset` concept in the C-API boundary. Therefore, this feature should not have breaking change in the C-API. Concretely, we add additional functions as `PGM_dataset_*_add_attribute_buffer` to allow user add columnar attribute buffers to the dataset. The user can call the dataset as below:

```c++
// create dataset
PGM_MutableDataset* dataset = PGM_create_dataset_mutable(handle, "input", 0, 0); 

// add row-based buffer for node
PGM_dataset_const_add_buffer(handle, dataset, "node", 5, 5, nullptr, node_buffer);

// add empty buffer for line by put nullptr in data, but with size definition
// decision made: do it this way, because using nullptr is a common way in C API to communicate that the buffer does not exist yet
PGM_dataset_const_add_buffer(handle, dataset, "line", 5, 5, nullptr, nullptr);
// and then add individual line attributes
PGM_dataset_const_add_attribute_buffer(handle, dataset, "line", "id", line_id_buffer);
PGM_dataset_const_add_attribute_buffer(handle, dataset, "line", "r1", line_r1_buffer);

// add row buffer for sym_load
PGM_dataset_const_add_buffer(handle, dataset, "sym_load", 5, 5, nullptr, sym_load_buffer);
// the following line should return error. It is not allowed to add attribute buffer if the row-buffer is set.
PGM_dataset_const_add_attribute_buffer(handle, dataset, "sym_load", "p_specified", p_buffer);

// use the dataset for further actions like initialize model, put into serialization, etc.
```

## Python API

In the Python API, four non-breaking changes are expected.

1. In all the places where a PGM dataset is expected, for each component, the user should be able to also supply either a numpy structured array (returned by `initialize_array`) or a dictionary of numpy homogeneous arrays (e.g. `{"id": [1, 2], "u_rated": [150e3, 10e3]}`).
2. In the calculate functions, the user can put desired components and/or attributes in `output_component_types`. The Python wrapper needs to decide whether to create a structured array or dictionary of homogeneous arrays per component. We need to figure how maintain backwards compatibility.
3. In the deserialization, user can specify if they want row- or column-based arrays in the returned dataset. See decision below for more information
4. In the serialization, if the user gives a dataset which are all column-based, the user has to provide a dataset type because there is no way to deduce it. As long as there one row-based array in the user-provided dataset, we can still deduce the dataset type.

Decision made on step 3 (deserialization): 
For deserialization we support either row or column based deserialization (function argument: Enum). If a user wants to deserialize to columnar data the default is to deserialize all data present. A user can give an Optional function argument to specify the desired components and attributes. In that case, deserialization + a filter (for the specific components and attributes) is happening. Let's call this Optional function argument `filter`. Make sure this behavior is documented well + document that providing a filter might result in loss of data.

## Make id optional for batch update dataset in columnar format

From the user's perspective, the user would definitely like to provide a columnar batch dataset in a way that the `id` is not provided for a certain component. In that case, it should be inferred that the elements where attributes are to be updated via columnar buffer are in the exact same sequence of the input data. This is a realistic use-case and will be appreciated by the user, to save the additional step to just assign the exactly the same `id` as in the input data. The following Python code should work:

```python
model = PowerGridModel(input_data=input_data)
result = model.calculate_power_flow(
    update_data={
        "sym_load": {"p_specified": np.random.randn(n_step, n_sym_load)}
    }
)
```

# Implementation Proposal

To make this feature possible, following implementation suggestions are proposed in the C++ core:

1. (done)Rename [`DatasetHandler`](https://github.com/PowerGridModel/power-grid-model/blob/3c9b5bbc01267338bf61bee86598c18909493edb/power_grid_model_c/power_grid_model/include/power_grid_model/auxiliary/dataset_handler.hpp) to `Dataset`.
2. (done)Remove old [`Dataset`](https://github.com/PowerGridModel/power-grid-model/blob/3c9b5bbc01267338bf61bee86598c18909493edb/power_grid_model_c/power_grid_model/include/power_grid_model/auxiliary/dataset.hpp).
3. In the new `Dataset`, add buffer control and iteration functionality. It can detect if a component buffer is row or column based, and in case of column based, generate temporary object to have the full struct for `MainModel` to consume.
4. (done)Rewrite `MainModel` to use the new `Dataset`. This also relates to #431.
6. In `Serializer`, it should directly read the row and column based buffer and serialize them to `msgpack` and `json`.
7. In `Deserializer`, it should write the attributes either in row- or column-based depending on what buffers are set in the `WritableDataset`.
8. Make id optional in update dataset: in the main core, we need to have special treatment in `is_update_independent` to make `id` as optional attribute in the batch update dataset.
    1. `is_update_independent` should be per component instead of the whole dataset. So we can allow individual `sequence` for each component.
    1. For a certain component, if the buffer is row-based
        1. If the `id` of the row-based buffer is not all `NaN`, we use the current logic to determine if the component is independent.
        1. If the `id` of the row-based buffer is all `NaN`
            1. If the buffer is not uniform, or the buffer is uniform but `elements_per_scenario` is not the same as the number of elements in the input data (in the model). An error should be raised.
            1. If the above check passes, we assume the component buffer is independent. And we generate a `sequence` from `0` to `n_comp` for this component. This will be consumed by the update function so the update function does not do `id` lookup.
    1. For a certain component, if the buffer is columnar, we do the following:
        1.  If `id` attribute buffer is provided **and** it is not all `NaN`, we look at `id` to judge if the component is independent or not. We do not need to create proxy stuff which is waste of time. Just directly look at `id` buffer.
        1.  If `id` attribute buffer is not provided **or** if the `id` is provided but they are all `NaN`:
            1. If the buffer is not uniform, or the buffer is uniform but `elements_per_scenario` is not the same as the number of elements in the input data (in the model). An error should be raised.
            1. If the above check passes, we assume the component buffer is independent. And we generate a `sequence` from `0` to `n_comp` for this component. This will be consumed by the update function so the update function does not do `id` lookup.



Discussion / hints:
This will make the PyArrow conversions also easier ( https://github.com/orgs/PowerGridModel/discussions/2 )

Relates to https://github.com/PowerGridModel/power-grid-model-io/issues/190 
### An alternative idea around python API Steps 2, 3:

Additional arguments to the deserialization functions 

Old proposal: 2 arguments: 
- dataset_filter as `set[ComponentType] | list[ComponentType] | None | dict[ComponentType, set[str] | list[str] | None]` 
- dataset_format as Enum: row_based or columnar

New: Only one additional argument: 

- dataset_filter as `set[ComponentType] | list[ComponentType] | None | ... | dict[ComponentType, set[str] | list[str] | None | ...]` 
PGM deduces dataset format from that data_filter.

Note: The difference in both option is the `...` option.

#### Example

```python
# All row based
data_filter = None

# All column based
data_filter = ...

# specific components row based
data_filter = ["node", "line"]
data_filter = {"node", "line"}

# specific components, mixed row-based and columnar data
data_filter = {
  "node": None,       # row-based
  "line": ["id", "p_from"],  # columnar with these attributes
  "transformer": ...         # columnar with all attributes # <--- (proposed) Ellipsis is a python built-in constant
} 
```

Implementation proposal step 7.ii:

> 7. Make id optional in update dataset: in the main core, we need to have special treatment in is_update_independent to make id as optional attribute in the batch update dataset. 
>   ii. For a certain component, if the buffer is row-based, we use the same logic as is now.

Loosening the requirement that `id` be present in columnar data means that the serialized data may not contain `id` anymore. If a columnar dataset without `id`s is serialized and then deserialized as row-based dataset, then the row-based dataset does not contain `id`s anymore either (all `na_ID`). According to your proposed logic, this is not allowed to be used as update data. This breaks the symmetry between row-based datasets and columnar datasets. I don't think we should do that, which means that the id-less lookup logic should also be updated for row-based datasets
> Implementation proposal step 7.ii:
> 
> > 7. Make id optional in update dataset: in the main core, we need to have special treatment in is_update_independent to make id as optional attribute in the batch update dataset.
> >    ii. For a certain component, if the buffer is row-based, we use the same logic as is now.
> 
> Loosening the requirement that `id` be present in columnar data means that the serialized data may not contain `id` anymore. If a columnar dataset without `id`s is serialized and then deserialized as row-based dataset, then the row-based dataset does not contain `id`s anymore either (all `na_ID`). According to your proposed logic, this is not allowed to be used as update data. This breaks the symmetry between row-based datasets and columnar datasets. I don't think we should do that, which means that the id-less lookup logic should also be updated for row-based datasets

Good point. I have adjusted the step proposal.
Continuing [this thread](https://github.com/PowerGridModel/power-grid-model/pull/799#issuecomment-2428374189) and [this thread](https://github.com/PowerGridModel/power-grid-model/pull/799#issuecomment-2428416850) from #799 for potential follow-up improvements to columnar data here because this issue is still alive and kicking. We can convert this to one or more new issues if necessary:

>> * Since we don't have a create buffer function for columnar buffers (since probably there is no value in it due to the nature of columnar data), we should document this somewhere as there is an "inconsistency" on how row and columnar data are managed at the C API level. Or at least those are my thought, it may lead to confusion from the user perspective.
>
> I've also been thinking about that, but I don't know where to put it. Do you have a good proposal?

Not sure yet, it probably makes sense to directly comment something about it under `PGM_create_buffer`, but it should probably be standalone somewhere else.

>> * Should we (at some point) make something like `is_columnar` available through the C API? It may not be needed, but given that we offer functions to check if something is batch and its size, it would probably go along with those (and may even provide value in the python side).
> 
> I think that makes sense, but maybe out of scope of `v1.10.0`. @TonyXiang8787 do you have any thoughts on this?


> > Implementation proposal step 7.ii:
> > > 7. Make id optional in update dataset: in the main core, we need to have special treatment in is_update_independent to make id as optional attribute in the batch update dataset.
> > >    ii. For a certain component, if the buffer is row-based, we use the same logic as is now.
> > 
> > 
> > Loosening the requirement that `id` be present in columnar data means that the serialized data may not contain `id` anymore. If a columnar dataset without `id`s is serialized and then deserialized as row-based dataset, then the row-based dataset does not contain `id`s anymore either (all `na_ID`). According to your proposed logic, this is not allowed to be used as update data. This breaks the symmetry between row-based datasets and columnar datasets. I don't think we should do that, which means that the id-less lookup logic should also be updated for row-based datasets
> 
> Good point. I have adjusted the step proposal.

no action items, but since there was a miscommunication within the team, I will add the full customer journey illustrated in https://github.com/PowerGridModel/power-grid-model/issues/548#issuecomment-2333399651 here:

1. a production environment that uses columnar data.
  a. to save on data, they do not use IDs in their update data
2. an error occurs in the production environment
  a. they catch this and the POD automatically creates debug input and update data using `json_serialize_to_file` (without any additional options) so that they have a repro case
  b. the update data now contains no IDs
3. the dev starts debugging.
  a. they create a model from the input data
  b. they use json_deserialize_from_file (without any additional options)
  c. they obtain a row-based dataset (not columnar)
  d. their loaded update data does not contain IDs
4. debugging in progress
   a. assuming row-based and columnar behave differently):
      1. they cannot reproduce the same error and dive into a rabbit-hole that is not what they want to debug
      2. we get a support request. we easily find the error, cause it's pretty obvious what's up
      3. they complain because they did not expect the two datasets to behave differently

   b. assuming row-based and columnar do behave the same):
      1. they can reproduce the error
      2. they find the problem and solve it by themselves


Fix patch:
diff --git a/docs/examples/Asymmetric Calculation Example.ipynb b/docs/examples/Asymmetric Calculation Example.ipynb
index 0b3175711..b8d627667 100644
--- a/docs/examples/Asymmetric Calculation Example.ipynb
+++ b/docs/examples/Asymmetric Calculation Example.ipynb
@@ -46,11 +46,7 @@
    "source": [
     "# some basic imports\n",
     "import numpy as np\n",
-    "import warnings\n",
-    "\n",
-    "with warnings.catch_warnings(action=\"ignore\", category=DeprecationWarning):\n",
-    "    # suppress warning about pyarrow as future required dependency\n",
-    "    import pandas as pd\n",
+    "import pandas as pd\n",
     "\n",
     "from power_grid_model import LoadGenType, ComponentType, DatasetType\n",
     "from power_grid_model import PowerGridModel, CalculationMethod, CalculationType, MeasuredTerminalType\n",
diff --git a/docs/examples/Make Test Dataset.ipynb b/docs/examples/Make Test Dataset.ipynb
index 1258600be..645b36abd 100644
--- a/docs/examples/Make Test Dataset.ipynb
+++ b/docs/examples/Make Test Dataset.ipynb
@@ -161,11 +161,7 @@
     "# first build the network\n",
     "\n",
     "import numpy as np\n",
-    "import warnings\n",
-    "\n",
-    "with warnings.catch_warnings(action=\"ignore\", category=DeprecationWarning):\n",
-    "    # suppress warning about pyarrow as future required dependency\n",
-    "    import pandas as pd\n",
+    "import pandas as pd\n",
     "\n",
     "from power_grid_model import LoadGenType, ComponentType, DatasetType\n",
     "from power_grid_model import PowerGridModel\n",
diff --git a/docs/examples/Power Flow Example.ipynb b/docs/examples/Power Flow Example.ipynb
index 251332de1..9662e4aaf 100644
--- a/docs/examples/Power Flow Example.ipynb
+++ b/docs/examples/Power Flow Example.ipynb
@@ -45,11 +45,7 @@
    "source": [
     "# some basic imports\n",
     "import numpy as np\n",
-    "import warnings\n",
-    "\n",
-    "with warnings.catch_warnings(action=\"ignore\", category=DeprecationWarning):\n",
-    "    # suppress warning about pyarrow as future required dependency\n",
-    "    import pandas as pd\n",
+    "import pandas as pd\n",
     "\n",
     "from power_grid_model import LoadGenType, ComponentType, DatasetType, ComponentAttributeFilterOptions\n",
     "from power_grid_model import PowerGridModel, CalculationMethod, CalculationType\n",
@@ -640,15 +636,40 @@
   },
   {
    "cell_type": "markdown",
-   "id": "fcc57354",
+   "id": "25a3f722",
    "metadata": {},
    "source": [
-    "Columnar data is also supported"
+    "If you are updating all components of the same type (uniformly) in the same order as in the input data, providing IDs is optional."
    ]
   },
   {
    "cell_type": "code",
    "execution_count": 15,
+   "id": "6a4d9802",
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "update_sym_load_no_id = initialize_array(DatasetType.update, ComponentType.sym_load, 2)\n",
+    "update_sym_load_no_id[\"p_specified\"] = [30e6, 15e6]  # change active power for both sym_loads\n",
+    "# leave reactive power the same, no need to specify\n",
+    "\n",
+    "update_line_no_id = initialize_array(DatasetType.update, ComponentType.line, 3)\n",
+    "update_line_no_id[\"from_status\"] = [0, 1, 1]  # switch off at from side for line #3\n",
+    "\n",
+    "update_data_no_id = {ComponentType.sym_load: update_sym_load_no_id, ComponentType.line: update_line_no_id}"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "id": "fcc57354",
+   "metadata": {},
+   "source": [
+    "Columnar data is also supported."
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 16,
    "id": "aaf8fe64",
    "metadata": {},
    "outputs": [],
@@ -660,7 +681,34 @@
     "}\n",
     "# leave to-side swiching status the same, no need to specify\n",
     "\n",
-    "update_data = {ComponentType.sym_load: update_sym_load, ComponentType.line: columnar_update_line}"
+    "update_data_col = {ComponentType.sym_load: update_sym_load, ComponentType.line: columnar_update_line}"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "id": "f5179288",
+   "metadata": {},
+   "source": [
+    "Columnar data also supports optional IDs."
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 17,
+   "id": "7eabed0c",
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "line_update_dtype = power_grid_meta_data[DatasetType.update][ComponentType.line].dtype\n",
+    "columnar_no_ID_update_line = {\n",
+    "    # Update IDs are not specified\n",
+    "    \"from_status\": np.array(\n",
+    "        [0, 1, 1], dtype=line_update_dtype[\"from_status\"]\n",
+    "    ),  # The update for the whole column needs to be specified\n",
+    "}\n",
+    "# leave to-side swiching status the same, no need to specify\n",
+    "\n",
+    "update_data_col_no_id = {ComponentType.sym_load: update_sym_load, ComponentType.line: columnar_no_ID_update_line}"
    ]
   },
   {
@@ -676,14 +724,21 @@
   },
   {
    "cell_type": "code",
-   "execution_count": 16,
+   "execution_count": 18,
    "id": "63ea4cc3",
    "metadata": {},
    "outputs": [],
    "source": [
     "from power_grid_model.validation import assert_valid_batch_data\n",
     "\n",
-    "assert_valid_batch_data(input_data=input_data, update_data=update_data, calculation_type=CalculationType.power_flow)"
+    "assert_valid_batch_data(input_data=input_data, update_data=update_data, calculation_type=CalculationType.power_flow)\n",
+    "assert_valid_batch_data(\n",
+    "    input_data=input_data, update_data=update_data_no_id, calculation_type=CalculationType.power_flow\n",
+    ")\n",
+    "assert_valid_batch_data(input_data=input_data, update_data=update_data_col, calculation_type=CalculationType.power_flow)\n",
+    "assert_valid_batch_data(\n",
+    "    input_data=input_data, update_data=update_data_col_no_id, calculation_type=CalculationType.power_flow\n",
+    ")"
    ]
   },
   {
@@ -699,7 +754,7 @@
   },
   {
    "cell_type": "code",
-   "execution_count": 17,
+   "execution_count": 19,
    "id": "188f6663",
    "metadata": {},
    "outputs": [],
@@ -721,7 +776,7 @@
   },
   {
    "cell_type": "code",
-   "execution_count": 18,
+   "execution_count": 20,
    "id": "a93c1e16",
    "metadata": {},
    "outputs": [
@@ -810,7 +865,7 @@
   },
   {
    "cell_type": "code",
-   "execution_count": 19,
+   "execution_count": 21,
    "id": "42d5cd8f",
    "metadata": {},
    "outputs": [],
@@ -844,7 +899,7 @@
   },
   {
    "cell_type": "code",
-   "execution_count": 20,
+   "execution_count": 22,
    "id": "4e25006f",
    "metadata": {
     "scrolled": true
@@ -887,7 +942,7 @@
   },
   {
    "cell_type": "code",
-   "execution_count": 21,
+   "execution_count": 23,
    "id": "afccf7a2",
    "metadata": {
     "scrolled": true
@@ -924,7 +979,7 @@
   },
   {
    "cell_type": "code",
-   "execution_count": 22,
+   "execution_count": 24,
    "id": "9af1be38",
    "metadata": {
     "scrolled": true
@@ -967,7 +1022,7 @@
   },
   {
    "cell_type": "code",
-   "execution_count": 23,
+   "execution_count": 25,
    "id": "041368dc",
    "metadata": {},
    "outputs": [],
@@ -997,7 +1052,7 @@
   },
   {
    "cell_type": "code",
-   "execution_count": 24,
+   "execution_count": 26,
    "id": "34338ce3",
    "metadata": {},
    "outputs": [
@@ -1029,7 +1084,7 @@
   },
   {
    "cell_type": "code",
-   "execution_count": 25,
+   "execution_count": 27,
    "id": "04e56690",
    "metadata": {},
    "outputs": [],
@@ -1061,7 +1116,7 @@
   },
   {
    "cell_type": "code",
-   "execution_count": 26,
+   "execution_count": 28,
    "id": "0d5b94c2",
    "metadata": {},
    "outputs": [
@@ -1144,7 +1199,7 @@
   },
   {
    "cell_type": "code",
-   "execution_count": 27,
+   "execution_count": 29,
    "id": "b5f10bae",
    "metadata": {},
    "outputs": [
@@ -1204,7 +1259,7 @@
   },
   {
    "cell_type": "code",
-   "execution_count": 28,
+   "execution_count": 30,
    "id": "1a221507",
    "metadata": {},
    "outputs": [
@@ -1246,7 +1301,7 @@
   },
   {
    "cell_type": "code",
-   "execution_count": 29,
+   "execution_count": 31,
    "id": "541af620",
    "metadata": {},
    "outputs": [
@@ -1294,7 +1349,7 @@
   },
   {
    "cell_type": "code",
-   "execution_count": 30,
+   "execution_count": 32,
    "id": "20d8285c",
    "metadata": {},
    "outputs": [],
@@ -1305,7 +1360,7 @@
   },
   {
    "cell_type": "code",
-   "execution_count": 31,
+   "execution_count": 33,
    "id": "b702eb15",
    "metadata": {},
    "outputs": [
@@ -1340,7 +1395,7 @@
   },
   {
    "cell_type": "code",
-   "execution_count": 32,
+   "execution_count": 34,
    "id": "1ba71901",
    "metadata": {},
    "outputs": [
@@ -1349,16 +1404,16 @@
      "output_type": "stream",
      "text": [
       "Node data with invalid results\n",
-      "[[9.99401170e-001 9.92685785e-001 9.94521366e-001]\n",
-      " [9.99347687e-001 9.86226389e-001 9.89352855e-001]\n",
-      " [9.99288384e-001 9.79654011e-001 9.84095542e-001]\n",
-      " [0.00000000e+000 3.04369633e+101 3.41345331e+241]\n",
-      " [9.99151380e-001 9.66149483e-001 9.73298790e-001]\n",
-      " [9.99073166e-001 9.59205860e-001 9.67750710e-001]\n",
-      " [9.98988099e-001 9.52126208e-001 9.62096474e-001]\n",
-      " [1.66994188e-321 3.12878333e-312 9.75772002e+199]\n",
-      " [9.98796126e-001 9.37530046e-001 9.50447962e-001]\n",
-      " [9.98688504e-001 9.29997471e-001 9.44441670e-001]]\n",
+      "[[0.99940117 0.99268579 0.99452137]\n",
+      " [0.99934769 0.98622639 0.98935286]\n",
+      " [0.99928838 0.97965401 0.98409554]\n",
+      " [0.         0.         0.        ]\n",
+      " [0.99915138 0.96614948 0.97329879]\n",
+      " [0.99907317 0.95920586 0.96775071]\n",
+      " [0.9989881  0.95212621 0.96209647]\n",
+      " [0.         0.         0.        ]\n",
+      " [0.99879613 0.93753005 0.95044796]\n",
+      " [0.9986885  0.92999747 0.94444167]]\n",
       "Node data with only valid results\n",
       "[[0.99940117 0.99268579 0.99452137]\n",
       " [0.99934769 0.98622639 0.98935286]\n",
diff --git a/docs/examples/Serialization Example.ipynb b/docs/examples/Serialization Example.ipynb
index b718946da..2353adc93 100644
--- a/docs/examples/Serialization Example.ipynb
+++ b/docs/examples/Serialization Example.ipynb
@@ -23,12 +23,8 @@
    "outputs": [],
    "source": [
     "import json\n",
+    "from pandas import DataFrame\n",
     "import pprint\n",
-    "import warnings\n",
-    "\n",
-    "with warnings.catch_warnings(action=\"ignore\", category=DeprecationWarning):\n",
-    "    # suppress warning about pyarrow as future required dependency\n",
-    "    from pandas import DataFrame\n",
     "\n",
     "from power_grid_model import PowerGridModel, ComponentType, ComponentAttributeFilterOptions\n",
     "from power_grid_model.utils import json_deserialize, json_serialize"
diff --git a/docs/examples/Short Circuit Example.ipynb b/docs/examples/Short Circuit Example.ipynb
index bf3479305..92d521c80 100644
--- a/docs/examples/Short Circuit Example.ipynb
+++ b/docs/examples/Short Circuit Example.ipynb
@@ -48,11 +48,7 @@
    "source": [
     "# some basic imports\n",
     "import numpy as np\n",
-    "import warnings\n",
-    "\n",
-    "with warnings.catch_warnings(action=\"ignore\", category=DeprecationWarning):\n",
-    "    # suppress warning about pyarrow as future required dependency\n",
-    "    import pandas as pd\n",
+    "import pandas as pd\n",
     "\n",
     "from power_grid_model import LoadGenType, ComponentType, DatasetType\n",
     "from power_grid_model import (\n",
diff --git a/docs/examples/State Estimation Example.ipynb b/docs/examples/State Estimation Example.ipynb
index 5066ac316..77a5007e4 100644
--- a/docs/examples/State Estimation Example.ipynb
+++ b/docs/examples/State Estimation Example.ipynb
@@ -43,11 +43,7 @@
    "source": [
     "# some basic imports\n",
     "import numpy as np\n",
-    "import warnings\n",
-    "\n",
-    "with warnings.catch_warnings(action=\"ignore\", category=DeprecationWarning):\n",
-    "    # suppress warning about pyarrow as future required dependency\n",
-    "    import pandas as pd\n",
+    "import pandas as pd\n",
     "\n",
     "from power_grid_model import LoadGenType, DatasetType, ComponentType\n",
     "from power_grid_model import PowerGridModel, CalculationMethod, CalculationType, MeasuredTerminalType\n",
diff --git a/docs/examples/Transformer Examples.ipynb b/docs/examples/Transformer Examples.ipynb
index e949cc0da..54a699504 100644
--- a/docs/examples/Transformer Examples.ipynb
+++ b/docs/examples/Transformer Examples.ipynb
@@ -38,12 +38,7 @@
    "source": [
     "# some basic imports\n",
     "import numpy as np\n",
-    "import warnings\n",
-    "\n",
-    "with warnings.catch_warnings():\n",
-    "    warnings.simplefilter(\"ignore\", category=DeprecationWarning)\n",
-    "    # suppress warning about pyarrow as future required dependency\n",
-    "    import pandas as pd\n",
+    "import pandas as pd\n",
     "\n",
     "from power_grid_model import LoadGenType, DatasetType, ComponentType\n",
     "from power_grid_model import (\n",
diff --git a/docs/examples/Validation Examples.ipynb b/docs/examples/Validation Examples.ipynb
index 6b09acf8c..3036a962e 100644
--- a/docs/examples/Validation Examples.ipynb
+++ b/docs/examples/Validation Examples.ipynb
@@ -110,12 +110,12 @@
      "evalue": "Wrong type for object with id 4\n\nTry validate_input_data() or validate_batch_data() to validate your data.\n",
      "output_type": "error",
      "traceback": [
-      "\u001b[1;31m---------------------------------------------------------------------------\u001b[0m",
-      "\u001b[1;31mIDWrongType\u001b[0m                               Traceback (most recent call last)",
-      "Cell \u001b[1;32mIn[2], line 2\u001b[0m\n\u001b[0;32m      1\u001b[0m \u001b[38;5;66;03m# Without validation\u001b[39;00m\n\u001b[1;32m----> 2\u001b[0m model \u001b[38;5;241m=\u001b[39m \u001b[43mPowerGridModel\u001b[49m\u001b[43m(\u001b[49m\u001b[43merror_data\u001b[49m\u001b[43m)\u001b[49m\n\u001b[0;32m      3\u001b[0m output_data \u001b[38;5;241m=\u001b[39m model\u001b[38;5;241m.\u001b[39mcalculate_state_estimation(symmetric\u001b[38;5;241m=\u001b[39m\u001b[38;5;28;01mTrue\u001b[39;00m)\n",
-      "File \u001b[1;32m~\\Documents\\power-grid-model\\src\\power_grid_model\\core\\power_grid_model.py:125\u001b[0m, in \u001b[0;36mPowerGridModel.__init__\u001b[1;34m(self, input_data, system_frequency)\u001b[0m\n\u001b[0;32m    123\u001b[0m prepared_input \u001b[38;5;241m=\u001b[39m prepare_input_view(_map_to_component_types(input_data))\n\u001b[0;32m    124\u001b[0m \u001b[38;5;28mself\u001b[39m\u001b[38;5;241m.\u001b[39m_model_ptr \u001b[38;5;241m=\u001b[39m pgc\u001b[38;5;241m.\u001b[39mcreate_model(system_frequency, input_data\u001b[38;5;241m=\u001b[39mprepared_input\u001b[38;5;241m.\u001b[39mget_dataset_ptr())\n\u001b[1;32m--> 125\u001b[0m \u001b[43massert_no_error\u001b[49m\u001b[43m(\u001b[49m\u001b[43m)\u001b[49m\n\u001b[0;32m    126\u001b[0m \u001b[38;5;28mself\u001b[39m\u001b[38;5;241m.\u001b[39m_all_component_count \u001b[38;5;241m=\u001b[39m {k: v \u001b[38;5;28;01mfor\u001b[39;00m k, v \u001b[38;5;129;01min\u001b[39;00m prepared_input\u001b[38;5;241m.\u001b[39mget_info()\u001b[38;5;241m.\u001b[39mtotal_elements()\u001b[38;5;241m.\u001b[39mitems() \u001b[38;5;28;01mif\u001b[39;00m v \u001b[38;5;241m>\u001b[39m \u001b[38;5;241m0\u001b[39m}\n",
-      "File \u001b[1;32m~\\Documents\\power-grid-model\\src\\power_grid_model\\core\\error_handling.py:169\u001b[0m, in \u001b[0;36massert_no_error\u001b[1;34m(batch_size, decode_error)\u001b[0m\n\u001b[0;32m    167\u001b[0m error \u001b[38;5;241m=\u001b[39m find_error(batch_size\u001b[38;5;241m=\u001b[39mbatch_size, decode_error\u001b[38;5;241m=\u001b[39mdecode_error)\n\u001b[0;32m    168\u001b[0m \u001b[38;5;28;01mif\u001b[39;00m error \u001b[38;5;129;01mis\u001b[39;00m \u001b[38;5;129;01mnot\u001b[39;00m \u001b[38;5;28;01mNone\u001b[39;00m:\n\u001b[1;32m--> 169\u001b[0m     \u001b[38;5;28;01mraise\u001b[39;00m error\n",
-      "\u001b[1;31mIDWrongType\u001b[0m: Wrong type for object with id 4\n\nTry validate_input_data() or validate_batch_data() to validate your data.\n"
+      "\u001b[0;31m---------------------------------------------------------------------------\u001b[0m",
+      "\u001b[0;31mIDWrongType\u001b[0m                               Traceback (most recent call last)",
+      "Cell \u001b[0;32mIn[2], line 2\u001b[0m\n\u001b[1;32m      1\u001b[0m \u001b[38;5;66;03m# Without validation\u001b[39;00m\n\u001b[0;32m----> 2\u001b[0m model \u001b[38;5;241m=\u001b[39m \u001b[43mPowerGridModel\u001b[49m\u001b[43m(\u001b[49m\u001b[43merror_data\u001b[49m\u001b[43m)\u001b[49m\n\u001b[1;32m      3\u001b[0m output_data \u001b[38;5;241m=\u001b[39m model\u001b[38;5;241m.\u001b[39mcalculate_state_estimation(symmetric\u001b[38;5;241m=\u001b[39m\u001b[38;5;28;01mTrue\u001b[39;00m)\n",
+      "File \u001b[0;32m~/pgm/power-grid-model/src/power_grid_model/core/power_grid_model.py:125\u001b[0m, in \u001b[0;36mPowerGridModel.__init__\u001b[0;34m(self, input_data, system_frequency)\u001b[0m\n\u001b[1;32m    123\u001b[0m prepared_input \u001b[38;5;241m=\u001b[39m prepare_input_view(_map_to_component_types(input_data))\n\u001b[1;32m    124\u001b[0m \u001b[38;5;28mself\u001b[39m\u001b[38;5;241m.\u001b[39m_model_ptr \u001b[38;5;241m=\u001b[39m pgc\u001b[38;5;241m.\u001b[39mcreate_model(system_frequency, input_data\u001b[38;5;241m=\u001b[39mprepared_input\u001b[38;5;241m.\u001b[39mget_dataset_ptr())\n\u001b[0;32m--> 125\u001b[0m \u001b[43massert_no_error\u001b[49m\u001b[43m(\u001b[49m\u001b[43m)\u001b[49m\n\u001b[1;32m    126\u001b[0m \u001b[38;5;28mself\u001b[39m\u001b[38;5;241m.\u001b[39m_all_component_count \u001b[38;5;241m=\u001b[39m {k: v \u001b[38;5;28;01mfor\u001b[39;00m k, v \u001b[38;5;129;01min\u001b[39;00m prepared_input\u001b[38;5;241m.\u001b[39mget_info()\u001b[38;5;241m.\u001b[39mtotal_elements()\u001b[38;5;241m.\u001b[39mitems() \u001b[38;5;28;01mif\u001b[39;00m v \u001b[38;5;241m>\u001b[39m \u001b[38;5;241m0\u001b[39m}\n",
+      "File \u001b[0;32m~/pgm/power-grid-model/src/power_grid_model/core/error_handling.py:169\u001b[0m, in \u001b[0;36massert_no_error\u001b[0;34m(batch_size, decode_error)\u001b[0m\n\u001b[1;32m    167\u001b[0m error \u001b[38;5;241m=\u001b[39m find_error(batch_size\u001b[38;5;241m=\u001b[39mbatch_size, decode_error\u001b[38;5;241m=\u001b[39mdecode_error)\n\u001b[1;32m    168\u001b[0m \u001b[38;5;28;01mif\u001b[39;00m error \u001b[38;5;129;01mis\u001b[39;00m \u001b[38;5;129;01mnot\u001b[39;00m \u001b[38;5;28;01mNone\u001b[39;00m:\n\u001b[0;32m--> 169\u001b[0m     \u001b[38;5;28;01mraise\u001b[39;00m error\n",
+      "\u001b[0;31mIDWrongType\u001b[0m: Wrong type for object with id 4\n\nTry validate_input_data() or validate_batch_data() to validate your data.\n"
      ]
     }
    ],
@@ -136,11 +136,11 @@
      "evalue": "There are 5 validation errors in input_data:\n   1. Fields line.id and sym_power_sensor.id are not unique for 2 lines/sym_power_sensors.\n   2. Field 'to_node' does not contain a valid node id for 1 line.\n   3. Field 'power_sigma' is not greater than zero for 2 sym_power_sensors.\n   4. Field 'measured_object' does not contain a valid line/generic_branch/transformer id for 1 sym_power_sensor. (measured_terminal_type=branch_from)\n   5. Field 'measured_object' does not contain a valid source id for 1 sym_power_sensor. (measured_terminal_type=source)",
      "output_type": "error",
      "traceback": [
-      "\u001b[1;31m---------------------------------------------------------------------------\u001b[0m",
-      "\u001b[1;31mValidationException\u001b[0m                       Traceback (most recent call last)",
-      "Cell \u001b[1;32mIn[3], line 4\u001b[0m\n\u001b[0;32m      1\u001b[0m \u001b[38;5;28;01mfrom\u001b[39;00m \u001b[38;5;21;01mpower_grid_model\u001b[39;00m\u001b[38;5;21;01m.\u001b[39;00m\u001b[38;5;21;01mvalidation\u001b[39;00m \u001b[38;5;28;01mimport\u001b[39;00m assert_valid_input_data\n\u001b[0;32m      3\u001b[0m \u001b[38;5;66;03m# Assert valid data\u001b[39;00m\n\u001b[1;32m----> 4\u001b[0m \u001b[43massert_valid_input_data\u001b[49m\u001b[43m(\u001b[49m\u001b[43merror_data\u001b[49m\u001b[43m,\u001b[49m\u001b[43m \u001b[49m\u001b[43msymmetric\u001b[49m\u001b[38;5;241;43m=\u001b[39;49m\u001b[38;5;28;43;01mTrue\u001b[39;49;00m\u001b[43m)\u001b[49m\n\u001b[0;32m      5\u001b[0m model \u001b[38;5;241m=\u001b[39m PowerGridModel(error_data)\n\u001b[0;32m      6\u001b[0m output_data \u001b[38;5;241m=\u001b[39m model\u001b[38;5;241m.\u001b[39mcalculate_state_estimation(symmetric\u001b[38;5;241m=\u001b[39m\u001b[38;5;28;01mTrue\u001b[39;00m)\n",
-      "File \u001b[1;32m~\\Documents\\power-grid-model\\src\\power_grid_model\\validation\\assertions.py:57\u001b[0m, in \u001b[0;36massert_valid_input_data\u001b[1;34m(input_data, calculation_type, symmetric)\u001b[0m\n\u001b[0;32m     53\u001b[0m validation_errors \u001b[38;5;241m=\u001b[39m validate_input_data(\n\u001b[0;32m     54\u001b[0m     input_data\u001b[38;5;241m=\u001b[39minput_data, calculation_type\u001b[38;5;241m=\u001b[39mcalculation_type, symmetric\u001b[38;5;241m=\u001b[39msymmetric\n\u001b[0;32m     55\u001b[0m )\n\u001b[0;32m     56\u001b[0m \u001b[38;5;28;01mif\u001b[39;00m validation_errors:\n\u001b[1;32m---> 57\u001b[0m     \u001b[38;5;28;01mraise\u001b[39;00m ValidationException(validation_errors, \u001b[38;5;124m\"\u001b[39m\u001b[38;5;124minput_data\u001b[39m\u001b[38;5;124m\"\u001b[39m)\n",
-      "\u001b[1;31mValidationException\u001b[0m: There are 5 validation errors in input_data:\n   1. Fields line.id and sym_power_sensor.id are not unique for 2 lines/sym_power_sensors.\n   2. Field 'to_node' does not contain a valid node id for 1 line.\n   3. Field 'power_sigma' is not greater than zero for 2 sym_power_sensors.\n   4. Field 'measured_object' does not contain a valid line/generic_branch/transformer id for 1 sym_power_sensor. (measured_terminal_type=branch_from)\n   5. Field 'measured_object' does not contain a valid source id for 1 sym_power_sensor. (measured_terminal_type=source)"
+      "\u001b[0;31m---------------------------------------------------------------------------\u001b[0m",
+      "\u001b[0;31mValidationException\u001b[0m                       Traceback (most recent call last)",
+      "Cell \u001b[0;32mIn[3], line 4\u001b[0m\n\u001b[1;32m      1\u001b[0m \u001b[38;5;28;01mfrom\u001b[39;00m \u001b[38;5;21;01mpower_grid_model\u001b[39;00m\u001b[38;5;21;01m.\u001b[39;00m\u001b[38;5;21;01mvalidation\u001b[39;00m \u001b[38;5;28;01mimport\u001b[39;00m assert_valid_input_data\n\u001b[1;32m      3\u001b[0m \u001b[38;5;66;03m# Assert valid data\u001b[39;00m\n\u001b[0;32m----> 4\u001b[0m \u001b[43massert_valid_input_data\u001b[49m\u001b[43m(\u001b[49m\u001b[43merror_data\u001b[49m\u001b[43m,\u001b[49m\u001b[43m \u001b[49m\u001b[43msymmetric\u001b[49m\u001b[38;5;241;43m=\u001b[39;49m\u001b[38;5;28;43;01mTrue\u001b[39;49;00m\u001b[43m)\u001b[49m\n\u001b[1;32m      5\u001b[0m model \u001b[38;5;241m=\u001b[39m PowerGridModel(error_data)\n\u001b[1;32m      6\u001b[0m output_data \u001b[38;5;241m=\u001b[39m model\u001b[38;5;241m.\u001b[39mcalculate_state_estimation(symmetric\u001b[38;5;241m=\u001b[39m\u001b[38;5;28;01mTrue\u001b[39;00m)\n",
+      "File \u001b[0;32m~/pgm/power-grid-model/src/power_grid_model/validation/assertions.py:57\u001b[0m, in \u001b[0;36massert_valid_input_data\u001b[0;34m(input_data, calculation_type, symmetric)\u001b[0m\n\u001b[1;32m     53\u001b[0m validation_errors \u001b[38;5;241m=\u001b[39m validate_input_data(\n\u001b[1;32m     54\u001b[0m     input_data\u001b[38;5;241m=\u001b[39minput_data, calculation_type\u001b[38;5;241m=\u001b[39mcalculation_type, symmetric\u001b[38;5;241m=\u001b[39msymmetric\n\u001b[1;32m     55\u001b[0m )\n\u001b[1;32m     56\u001b[0m \u001b[38;5;28;01mif\u001b[39;00m validation_errors:\n\u001b[0;32m---> 57\u001b[0m     \u001b[38;5;28;01mraise\u001b[39;00m ValidationException(validation_errors, \u001b[38;5;124m\"\u001b[39m\u001b[38;5;124minput_data\u001b[39m\u001b[38;5;124m\"\u001b[39m)\n",
+      "\u001b[0;31mValidationException\u001b[0m: There are 5 validation errors in input_data:\n   1. Fields line.id and sym_power_sensor.id are not unique for 2 lines/sym_power_sensors.\n   2. Field 'to_node' does not contain a valid node id for 1 line.\n   3. Field 'power_sigma' is not greater than zero for 2 sym_power_sensors.\n   4. Field 'measured_object' does not contain a valid line/generic_branch/transformer id for 1 sym_power_sensor. (measured_terminal_type=branch_from)\n   5. Field 'measured_object' does not contain a valid source id for 1 sym_power_sensor. (measured_terminal_type=source)"
      ]
     }
    ],
@@ -233,11 +233,11 @@
       "\t\tids: [(<ComponentType.line: 'line'>, np.int32(6)), (<ComponentType.sym_power_sensor: 'sym_power_sensor'>, np.int32(6))]\n",
       "\n",
       "\tField 'to_node' does not contain a valid node id for 1 line.\n",
-      "\t\tcomponent: line\n",
-      "\t\tfield: 'to_node'\n",
       "\t\tids: [6]\n",
       "\t\tref_components: node\n",
       "\t\tfilters: \n",
+      "\t\tcomponent: line\n",
+      "\t\tfield: 'to_node'\n",
       "\n",
       "\tField 'power_sigma' is not greater than zero for 2 sym_power_sensors.\n",
       "\t\tcomponent: sym_power_sensor\n",
@@ -246,18 +246,18 @@
       "\t\tref_value: zero\n",
       "\n",
       "\tField 'measured_object' does not contain a valid line/generic_branch/transformer id for 1 sym_power_sensor. (measured_terminal_type=branch_from)\n",
-      "\t\tcomponent: sym_power_sensor\n",
-      "\t\tfield: 'measured_object'\n",
       "\t\tids: [6]\n",
       "\t\tref_components: line/generic_branch/transformer\n",
       "\t\tfilters: (measured_terminal_type=branch_from)\n",
-      "\n",
-      "\tField 'measured_object' does not contain a valid source id for 1 sym_power_sensor. (measured_terminal_type=source)\n",
       "\t\tcomponent: sym_power_sensor\n",
       "\t\tfield: 'measured_object'\n",
+      "\n",
+      "\tField 'measured_object' does not contain a valid source id for 1 sym_power_sensor. (measured_terminal_type=source)\n",
       "\t\tids: [7]\n",
       "\t\tref_components: source\n",
       "\t\tfilters: (measured_terminal_type=source)\n",
+      "\t\tcomponent: sym_power_sensor\n",
+      "\t\tfield: 'measured_object'\n",
       "\n"
      ]
     }
diff --git a/docs/user_manual/components.md b/docs/user_manual/components.md
index a32869699..4a5730b1e 100644
--- a/docs/user_manual/components.md
+++ b/docs/user_manual/components.md
@@ -18,7 +18,11 @@ The base type for all power-grid-model components.
 
 | name | data type | unit | description                                                                                                                                                       | required |                                     update                                     |
 | ---- | --------- | ---- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------- | :------: | :----------------------------------------------------------------------------: |
-| `id` | `int32_t` | -    | ID of a component. The ID should be unique across all components within the same scenario, e.g., you cannot have a node with `id=5` and another line with `id=5`. | &#10004; | &#10060; (id needs to be specified in the update query, but cannot be changed) |
+| `id` | `int32_t` | -    | ID of a component. The ID should be unique across all components within the same scenario, e.g., you cannot have a node with `id=5` and another line with `id=5`. | &#10004; | &#10060; (see below) |
+
+If a component update is uniform and is updating all the elements with the same component type, IDs can be omitted or set to `nan`s. In any other case, the IDs need to be present in the update dataset, but cannot be changed.
+
+Uniform component updates are ones that update the same number of component of the same type across scenarios. By definition, a dense update is always uniform, hence the IDs can always be optional; whereas for a sparse update, the IDs can only be optional if the index pointer is given as an arithmetic sequence of all elements (i.e. a sparse representation of a dense buffer). An example of the usage of optional IDs is given in [Power Flow Example](./Power%20Flow%20Example.ipynb)
 
 #### Steady state output and Short circuit output
 


Test patch: