Repository: green-coding-solutions/green-metrics-tool
Pull request #1189: Splits the diskio provider into reads and writes
Author: login: ribalba
Created: 2025-05-14T15:45:28Z | Merged: 2025-05-18T15:01:11Z

<!-- greptile_comment -->

## Greptile Summary

This PR splits disk I/O metrics into separate read and write measurements, requiring significant changes across the codebase to support multiple metrics per timestamp.

- Identified duplicate `_parse_metrics()` methods in `/metric_providers/disk/io/procfs/system/provider.py` that need to be resolved
- The `DiskIoParseMixin` in `/metric_providers/disk/io/disk_io_parse.py` is tightly coupled with hardcoded names and needs refactoring for better reusability
- Missing test coverage for the new disk I/O split functionality and mixin implementation
- The removal of sector-level error checking in procfs provider could mask underlying issues
- The database schema change to remove unique index on `measurement_values` needs careful consideration for data integrity



<!-- /greptile_comment -->


Diff:
diff --git a/docker/structure.sql b/docker/structure.sql
index 688f24345..8b37add7e 100644
--- a/docker/structure.sql
+++ b/docker/structure.sql
@@ -273,7 +273,6 @@ CREATE TABLE measurement_values (
 CREATE INDEX measurement_values_mmid ON measurement_values(measurement_metric_id);
 CREATE UNIQUE INDEX measurement_values_unique ON measurement_values(measurement_metric_id, time);
 
-
 CREATE TABLE network_intercepts (
     id SERIAL PRIMARY KEY,
     run_id uuid NOT NULL REFERENCES runs(id) ON DELETE CASCADE ON UPDATE CASCADE ,
diff --git a/frontend/js/helpers/config.js.example b/frontend/js/helpers/config.js.example
index 49cde4e3a..f56fe555b 100644
--- a/frontend/js/helpers/config.js.example
+++ b/frontend/js/helpers/config.js.example
@@ -116,10 +116,15 @@ METRIC_MAPPINGS = {
         "source": "statvfs syscall",
         "explanation": "Disk used space of the root filesystem via statvfs syscall"
     },
-    "disk_io_procfs_system": {
-        "clean_name": "Disk I/O",
+    "disk_io_read_procfs_system": {
+        "clean_name": "Disk I/O (Read)",
         "source": "procfs",
-        "explanation": "Disk I/O for whole system via procfs"
+        "explanation": "Disk bytes read for whole system via procfs"
+    },
+    "disk_io_write_procfs_system": {
+        "clean_name": "Disk I/O (Write)",
+        "source": "procfs",
+        "explanation": "Disk bytes written for whole system via procfs"
     },
     "disk_io_cgroup_container": {
         "clean_name": "Disk I/O",
@@ -141,10 +146,15 @@ METRIC_MAPPINGS = {
         "source": "formula",
         "explanation": "Disk I/O Totals accumulated per container via cgroup"
     },
-    "disk_total_procfs_system": {
-        "clean_name": "Disk Data",
+    "disk_total_read_procfs_system": {
+        "clean_name": "Disk Data (Read)",
         "source": "formula",
-        "explanation": "Disk I/O Totals for whole system accumulated via procfs"
+        "explanation": "Disk read bytes totals for whole system accumulated via procfs"
+    },
+    "disk_total_write_procfs_system": {
+        "clean_name": "Disk Data (Write)",
+        "source": "formula",
+        "explanation": "Disk written bytes totals for whole system accumulated via procfs"
     },
     "psu_energy_cgroup_container": {
         "clean_name": "Container Energy",
@@ -455,5 +465,45 @@ METRIC_MAPPINGS = {
         "clean_name": "Machine Power",
         "source": "SDIA",
         "explanation": "Full machine power (AC) as estimated by SDIA model"
-    }
+    },
+    "disk_io_read_cgroup_container": {
+        "clean_name": "Disk Reads",
+        "source": "cgroup",
+        "explanation": "Data read from disk per container via cgroup"
+    },
+    "disk_io_write_cgroup_container": {
+        "clean_name": "Disk Writes",
+        "source": "cgroup",
+        "explanation": "Data written to disk per container via cgroup"
+    },
+    "disk_io_read_cgroup_system": {
+        "clean_name": "Disk Reads",
+        "source": "cgroup",
+        "explanation": "Data read from disk for the system via cgroup"
+    },
+    "disk_io_write_cgroup_system": {
+        "clean_name": "Disk Writes",
+        "source": "cgroup",
+        "explanation": "Data written to disk for the system via cgroup"
+    },
+    "disk_total_read_cgroup_container": {
+        "clean_name": "Total Disk Reads",
+        "source": "cgroup",
+        "explanation": "Total data read from disk per container via cgroup"
+    },
+    "disk_total_write_cgroup_container": {
+        "clean_name": "Total Disk Writes",
+        "source": "cgroup",
+        "explanation": "Total data written to disk per container via cgroup"
+    },
+    "disk_total_read_cgroup_system": {
+        "clean_name": "Total System Disk Reads",
+        "source": "cgroup",
+        "explanation": "Total data read from disk for the system via cgroup"
+    },
+    "disk_total_write_cgroup_system": {
+        "clean_name": "Total System Disk Writes",
+        "source": "cgroup",
+        "explanation": "Total data written to disk for the system via cgroup"
+    },
 } // PLEASE DO NOT REMOVE THIS COMMENT -- END METRIC_MAPPINGS
diff --git a/lib/phase_stats.py b/lib/phase_stats.py
index 52c1cd09e..a94dd3228 100644
--- a/lib/phase_stats.py
+++ b/lib/phase_stats.py
@@ -197,7 +197,19 @@ def build_and_store_phase_stats(run_id, sci=None):
                 if metric in ('cpu_utilization_cgroup_container', 'cpu_utilization_cgroup_system', ):
                     cpu_utilization_containers[detail_name] = avg_value
 
-            elif metric in ['network_io_cgroup_system', 'disk_io_cgroup_system', 'network_io_cgroup_container', 'network_io_procfs_system', 'disk_io_procfs_system', 'disk_io_cgroup_container', 'disk_io_bytesread_powermetrics_vm', 'disk_io_byteswritten_powermetrics_vm']:
+            elif metric in ['network_io_cgroup_system',
+                            'network_io_cgroup_container',
+                            'network_io_procfs_system',
+                            'disk_io_read_procfs_system',
+                            'disk_io_write_procfs_system',
+                            'disk_io_cgroup_container',
+                            'disk_io_bytesread_powermetrics_vm',
+                            'disk_io_byteswritten_powermetrics_vm',
+                            'disk_io_read_cgroup_container',
+                            'disk_io_write_cgroup_container',
+                            'disk_io_write_cgroup_system',
+                            'disk_io_read_cgroup_system',
+                            ]:
 
                 # if we only have one value, we cannot determine the effective sampling rate.
                 # Thus we have to use the configured sampling_rate from the settings
diff --git a/lib/scenario_runner.py b/lib/scenario_runner.py
index 4aad01647..29b151497 100644
--- a/lib/scenario_runner.py
+++ b/lib/scenario_runner.py
@@ -1484,7 +1484,11 @@ def stop_metric_providers(self):
                 errors.append(f"{metric_provider.__class__.__name__} returned error message: {str(exc)}")
                 continue
 
-            metric_importer.import_measurements(df, metric_provider._metric_name, self._run_id)
+            if isinstance(df, list):
+                for i, dfi in enumerate(df):
+                    metric_importer.import_measurements(dfi, metric_provider._sub_metrics_name[i], self._run_id)
+            else:
+                metric_importer.import_measurements(df, metric_provider._metric_name, self._run_id)
 
             print('Imported', TerminalColors.HEADER, len(df), TerminalColors.ENDC, 'metrics from ', metric_provider.__class__.__name__)
 
diff --git a/metric_providers/base.py b/metric_providers/base.py
index b5394f8a6..eda8524d7 100644
--- a/metric_providers/base.py
+++ b/metric_providers/base.py
@@ -152,8 +152,10 @@ def _add_and_validate_sampling_rate_and_jitter(self, df):
         return df
 
     def _add_auxiliary_fields(self, df): # can be overriden in child
-        df['unit'] = self._unit
-        df['metric'] = self._metric_name
+        if 'unit' not in df.columns:
+            df['unit'] = self._unit
+        if 'metric' not in df.columns:
+            df['metric'] = self._metric_name
         df['sampling_rate_configured'] = self._sampling_rate
         return df
 
@@ -168,13 +170,14 @@ def read_metrics(self): # should not be overriden
         self._check_resolution_underflow(df)
 
         df = self._parse_metrics(df)
-        df = self._add_auxiliary_fields(df)
 
-        df = self._add_and_validate_sampling_rate_and_jitter(df)
+        def process_df(df):
+            df = self._add_auxiliary_fields(df)
+            df = self._add_and_validate_sampling_rate_and_jitter(df)
+            self._check_empty(df)
+            return df
 
-        self._check_empty(df) # we do another check after transformations, as this could have resulted in zero rows
-
-        return df
+        return process_df(df) if not isinstance(df, list) else [process_df(dfi) for dfi in df]
 
     def _add_extra_switches(self, call_string): # will be adapted in child if needed
         return call_string
diff --git a/metric_providers/disk/io/cgroup/container/provider.py b/metric_providers/disk/io/cgroup/container/provider.py
index 06239f11b..8e441c622 100644
--- a/metric_providers/disk/io/cgroup/container/provider.py
+++ b/metric_providers/disk/io/cgroup/container/provider.py
@@ -1,9 +1,9 @@
 import os
 
-from lib import utils
 from metric_providers.container import ContainerMetricProvider
+from metric_providers.disk.io.disk_io_parse import DiskIoParseMixin
 
-class DiskIoCgroupContainerProvider(ContainerMetricProvider):
+class DiskIoCgroupContainerProvider(DiskIoParseMixin, ContainerMetricProvider):
     def __init__(self, sampling_rate, skip_check=False, containers: dict = None):
         super().__init__(
             metric_name='disk_io_cgroup_container',
@@ -14,30 +14,8 @@ def __init__(self, sampling_rate, skip_check=False, containers: dict = None):
             skip_check=skip_check,
             containers=containers,
         )
+        self._sub_metrics_name = ['disk_io_read_cgroup_container', 'disk_io_write_cgroup_container']
 
     def _parse_metrics(self, df):
         df = super()._parse_metrics(df)
-
-        df = df.sort_values(by=['detail_name', 'time'], ascending=True)
-
-        df['written_bytes_intervals'] = df.groupby(['detail_name'])['written_bytes'].diff()
-        df['written_bytes_intervals'] = df.groupby('detail_name')['written_bytes_intervals'].transform(utils.df_fill_mean) # fill first NaN value resulted from diff()
-
-        df['read_bytes_intervals'] = df.groupby(['detail_name'])['read_bytes'].diff()
-        df['read_bytes_intervals'] = df.groupby('detail_name')['read_bytes_intervals'].transform(utils.df_fill_mean) # fill first NaN value resulted from diff()
-
-        # we checked at ingest if it contains NA values. So NA can only occur if group diff resulted in only one value.
-        # Since one value is useless for us we drop the row
-        df.dropna(inplace=True)
-
-        if (df['read_bytes_intervals'] < 0).any():
-            raise ValueError('DiskIoCgroupContainerProvider data column read_bytes_intervals had negative values.')
-
-        if (df['written_bytes_intervals'] < 0).any():
-            raise ValueError('DiskIoCgroupContainerProvider data column written_bytes_intervals had negative values.')
-
-        df['value'] = df['read_bytes_intervals'] + df['written_bytes_intervals']
-        df['value'] = df.value.astype(int)
-        df = df.drop(columns=['read_bytes','written_bytes', 'written_bytes_intervals', 'read_bytes_intervals'])  # clean up
-
-        return df
+        return self._parse_metrics_splitup_helper(df)
diff --git a/metric_providers/disk/io/cgroup/system/provider.py b/metric_providers/disk/io/cgroup/system/provider.py
index 0e51a0dd0..d2b1685fa 100644
--- a/metric_providers/disk/io/cgroup/system/provider.py
+++ b/metric_providers/disk/io/cgroup/system/provider.py
@@ -1,8 +1,9 @@
 import os
 
 from metric_providers.cgroup import CgroupMetricProvider
+from metric_providers.disk.io.disk_io_parse import DiskIoParseMixin
 
-class DiskIoCgroupSystemProvider(CgroupMetricProvider):
+class DiskIoCgroupSystemProvider(DiskIoParseMixin, CgroupMetricProvider):
     def __init__(self, sampling_rate, skip_check=False, cgroups: dict = None):
         super().__init__(
             metric_name='disk_io_cgroup_system',
@@ -13,3 +14,9 @@ def __init__(self, sampling_rate, skip_check=False, cgroups: dict = None):
             skip_check=skip_check,
             cgroups=cgroups,
         )
+
+        self._sub_metrics_name = ['disk_io_read_cgroup_system', 'disk_io_write_cgroup_system']
+
+    def _parse_metrics(self, df):
+        df = super()._parse_metrics(df)
+        return self._parse_metrics_splitup_helper(df)
diff --git a/metric_providers/disk/io/disk_io_parse.py b/metric_providers/disk/io/disk_io_parse.py
new file mode 100644
index 000000000..4765af306
--- /dev/null
+++ b/metric_providers/disk/io/disk_io_parse.py
@@ -0,0 +1,44 @@
+from lib import utils
+
+class DiskIoParseMixin:
+    def _parse_metrics_splitup_helper(self, df):
+
+        df = df.sort_values(by=['detail_name', 'time'], ascending=True)
+
+        df['written_bytes_intervals'] = df.groupby(['detail_name'])['written_bytes'].diff()
+        df['written_bytes_intervals'] = df.groupby('detail_name')['written_bytes_intervals'].transform(utils.df_fill_mean) # fill first NaN value resulted from diff()
+
+        df['read_bytes_intervals'] = df.groupby(['detail_name'])['read_bytes'].diff()
+        df['read_bytes_intervals'] = df.groupby('detail_name')['read_bytes_intervals'].transform(utils.df_fill_mean) # fill first NaN value resulted from diff()
+
+        # we checked at ingest if it contains NA values. So NA can only occur if group diff resulted in only one value.
+        # Since one value is useless for us we drop the row
+        df.dropna(inplace=True)
+
+        if (df['read_bytes_intervals'] < 0).any():
+            raise ValueError(f"{self.__class__.__name__} data column read_bytes_intervals had negative values.")
+
+        if (df['written_bytes_intervals'] < 0).any():
+            raise ValueError(f"{self.__class__.__name__} data column written_bytes_intervals had negative values.")
+
+        base_cols = ['time', 'detail_name']
+
+        df_read = (
+            df[base_cols + ["read_bytes_intervals"]]
+            .rename(columns={"read_bytes_intervals": "value"})
+            .copy()
+        )
+        df_read["value"] = df_read["value"].astype(int)
+        df_read['unit'] = self._unit
+        df_read['metric'] = self._sub_metrics_name[0]
+
+        df_written = (
+            df[base_cols + ["written_bytes_intervals"]]
+            .rename(columns={"written_bytes_intervals": "value"})
+            .copy()
+        )
+        df_written["value"] = df_written["value"].astype(int)
+        df_written['unit'] = self._unit
+        df_written['metric'] = self._sub_metrics_name[1]
+
+        return [df_read, df_written]
diff --git a/metric_providers/disk/io/procfs/system/provider.py b/metric_providers/disk/io/procfs/system/provider.py
index 450eec616..0b882bc4d 100644
--- a/metric_providers/disk/io/procfs/system/provider.py
+++ b/metric_providers/disk/io/procfs/system/provider.py
@@ -1,10 +1,10 @@
 import os
 from functools import lru_cache
 
-from lib import utils
 from metric_providers.base import BaseMetricProvider
+from metric_providers.disk.io.disk_io_parse import DiskIoParseMixin
 
-class DiskIoProcfsSystemProvider(BaseMetricProvider):
+class DiskIoProcfsSystemProvider(BaseMetricProvider, DiskIoParseMixin):
     def __init__(self, sampling_rate, skip_check=False):
         super().__init__(
             metric_name='disk_io_procfs_system',
@@ -15,33 +15,21 @@ def __init__(self, sampling_rate, skip_check=False):
             skip_check=skip_check,
         )
 
-    def _parse_metrics(self, df):
-        df['detail_name'] = df['device']
-
-        df = df.sort_values(by=['device', 'time'], ascending=True)
-
-        df['read_sectors_intervals'] = df.groupby(['device'])['read_sectors'].diff()
-        df['read_sectors_intervals'] = df.groupby('device')['read_sectors_intervals'].transform(utils.df_fill_mean) # fill first NaN value resulted from diff()
-
-        df['written_sectors_intervals'] = df.groupby(['device'])['written_sectors'].diff()
-        df['written_sectors_intervals'] = df.groupby('device')['written_sectors_intervals'].transform(utils.df_fill_mean) # fill first NaN value resulted from diff()
+        self._sub_metrics_name = ['disk_io_read_procfs_system', 'disk_io_write_procfs_system']
 
-        # we checked at ingest if it contains NA values. So NA can only occur if group diff resulted in only one value.
-        # Since one value is useless for us we drop the row
-        df.dropna(inplace=True)
-
-        if (df['written_sectors_intervals'] < 0).any():
-            raise ValueError('DiskIoProcfsSystemProvider data column written_sectors_intervals had negative values.')
+    def _parse_metrics(self, df):
+        df = super()._parse_metrics(df)
+        return self._parse_metrics_splitup_helper(df)
 
-        if (df['read_sectors_intervals'] < 0).any():
-            raise ValueError('DiskIoProcfsSystemProvider data column read_sectors_intervals had negative values.')
+    def _parse_metrics(self, df):
+        df = super()._parse_metrics(df)
 
         df['blocksize'] = df['device'].apply(self.get_blocksize)
-        df['value'] = (df['read_sectors_intervals'] + df['written_sectors_intervals'])*df['blocksize']
-        df['value'] = df.value.astype(int)
-        df = df.drop(columns=['read_sectors','written_sectors', 'written_sectors_intervals', 'read_sectors_intervals', 'device', 'blocksize'])  # clean up
+        df['read_bytes'] = df['read_sectors']*df['blocksize']
+        df['written_bytes'] = df['written_sectors']*df['blocksize']
+        df['detail_name'] = df['device']
 
-        return df
+        return self._parse_metrics_splitup_helper(df)
 
     @lru_cache(maxsize=100)
     def get_blocksize(self, device_name):
diff --git a/migrations/2024_12_24_measurement_table_split.sql b/migrations/2024_12_24_measurement_table_split.sql
index 088e1b20f..ba8394bf1 100644
--- a/migrations/2024_12_24_measurement_table_split.sql
+++ b/migrations/2024_12_24_measurement_table_split.sql
@@ -28,15 +28,15 @@ SELECT DISTINCT run_id, metric, detail_name, unit
 FROM measurements;
 
 INSERT INTO measurement_values (measurement_metric_id, value, time)
-SELECT 
+SELECT
     mm.id AS measurement_metric_id,
     m.value,
     m.time
-FROM 
+FROM
     measurements m
-JOIN 
+JOIN
     measurement_metrics mm
-ON 
+ON
     m.run_id = mm.run_id
     AND m.metric = mm.metric
     AND m.detail_name = mm.detail_name


Reviews:
user: login: greptile-apps[bot]
body: <sub>10 file(s) reviewed, 4 comment(s)</sub>
<sub>[Edit PR Review Bot Settings](https://app.greptile.com/review/github) | [Greptile](https://greptile.com?utm_source=greptile_expert&utm_medium=github&utm_campaign=code_reviews&utm_content=green-metrics-tool_1189)</sub>
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-05-15T09:58:29Z
user: login: ArneTR
body: I like the implementation. Suprisingly I think I would have made it through overloading the `detail_name`. This would have been far easier but would garble the column.
So although more work this implementation feels cleaner to me.

What is however not so future proof is the mixin I feel. It is very undynamic referencing self._sub_metrics_name[0]
 as implied knowledge and having many hardcoded names.
I would like to bring the functionality also to the NetworkIO reporter. Then it would have to be copied almost in full or a refactoring is needed.

Also: No tests?

All in all I would say good for a merge as we need the feature. But tests and refactoring should be prioritized for the next weeks to not have lingering legacy code here. 
state: CHANGES_REQUESTED
author_association: MEMBER
submitted_at: 2025-05-15T11:21:09Z
user: login: ribalba
state: COMMENTED
author_association: MEMBER
submitted_at: 2025-05-16T07:40:41Z
user: login: ribalba
state: COMMENTED
author_association: MEMBER
submitted_at: 2025-05-16T07:41:41Z
user: login: ribalba
state: COMMENTED
author_association: MEMBER
submitted_at: 2025-05-16T07:48:51Z
user: login: ArneTR
state: COMMENTED
author_association: MEMBER
submitted_at: 2025-05-16T09:08:20Z
user: login: ArneTR
state: COMMENTED
author_association: MEMBER
submitted_at: 2025-05-16T09:09:12Z
user: login: greptile-apps[bot]
body: <sub>10 file(s) reviewed, 4 comment(s)</sub>
<sub>[Edit PR Review Bot Settings](https://app.greptile.com/review/github) | [Greptile](https://greptile.com?utm_source=greptile_expert&utm_medium=github&utm_campaign=code_reviews&utm_content=green-metrics-tool_1189)</sub>
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-05-18T14:56:05Z

Review comments:
diff_hunk: @@ -454,5 +454,45 @@ METRIC_MAPPINGS = {
         "clean_name": "Machine Power",
         "source": "SDIA",
         "explanation": "Full machine power (AC) as estimated by SDIA model"
-    }
+    },
+    "disk_io_read_cgroup_container": {
+        "clean_name": "Disk Reads",
+        "source": "cgroup",
+        "explanation": "Data read from disk per container via cgroup"
+    },
+    "disk_io_write_cgroup_container": {
+        "clean_name": "Disk Writes",
+        "source": "cgroup",
+        "explanation": "Data written to disk per container via cgroup"
+    },
+    "disk_io_read_cgroup_system": {
+        "clean_name": "Disk Reads",
+        "source": "cgroup",
+        "explanation": "Data read from disk per container via cgroup"
+    },
path: frontend/js/helpers/config.js.example
user: login: greptile-apps[bot]
body: logic: Incorrect explanation - states 'per container' but this is a system metric

```suggestion
        "clean_name": "Disk Reads",
        "source": "cgroup",
        "explanation": "Data read from disk for the system via cgroup"
    },
```
created_at: 2025-05-15T09:57:13Z
author_association: CONTRIBUTOR
diff_hunk: @@ -454,5 +454,45 @@ METRIC_MAPPINGS = {
         "clean_name": "Machine Power",
         "source": "SDIA",
         "explanation": "Full machine power (AC) as estimated by SDIA model"
-    }
+    },
+    "disk_io_read_cgroup_container": {
+        "clean_name": "Disk Reads",
+        "source": "cgroup",
+        "explanation": "Data read from disk per container via cgroup"
+    },
+    "disk_io_write_cgroup_container": {
+        "clean_name": "Disk Writes",
+        "source": "cgroup",
+        "explanation": "Data written to disk per container via cgroup"
+    },
+    "disk_io_read_cgroup_system": {
+        "clean_name": "Disk Reads",
+        "source": "cgroup",
+        "explanation": "Data read from disk per container via cgroup"
+    },
+    "disk_io_write_cgroup_system": {
+        "clean_name": "Disk Writes",
+        "source": "cgroup",
+        "explanation": "Data written to disk per container via cgroup"
+    },
path: frontend/js/helpers/config.js.example
user: login: greptile-apps[bot]
body: logic: Incorrect explanation - states 'per container' but this is a system metric

```suggestion
        "source": "cgroup",
        "explanation": "Data written to disk for the system via cgroup"
    },
```
created_at: 2025-05-15T09:57:14Z
author_association: CONTRIBUTOR
diff_hunk: @@ -454,5 +454,45 @@ METRIC_MAPPINGS = {
         "clean_name": "Machine Power",
         "source": "SDIA",
         "explanation": "Full machine power (AC) as estimated by SDIA model"
-    }
+    },
+    "disk_io_read_cgroup_container": {
+        "clean_name": "Disk Reads",
+        "source": "cgroup",
+        "explanation": "Data read from disk per container via cgroup"
+    },
+    "disk_io_write_cgroup_container": {
+        "clean_name": "Disk Writes",
+        "source": "cgroup",
+        "explanation": "Data written to disk per container via cgroup"
+    },
+    "disk_io_read_cgroup_system": {
+        "clean_name": "Disk Reads",
+        "source": "cgroup",
+        "explanation": "Data read from disk per container via cgroup"
+    },
+    "disk_io_write_cgroup_system": {
+        "clean_name": "Disk Writes",
+        "source": "cgroup",
+        "explanation": "Data written to disk per container via cgroup"
+    },
+    "disk_total_read_cgroup_container": {
+        "clean_name": "Total Disk Reads",
+        "source": "cgroup",
+        "explanation": "Total data read from disk per container via cgroup"
+    },
+    "disk_total_write_cgroup_container": {
+        "clean_name": "Total Disk Writes",
+        "source": "cgroup",
+        "explanation": "Total data written to disk per container via cgroup"
+    },
+    "disk_total_read_cgroup_system": {
+        "clean_name": "Total Sytem Disk Reads",
path: frontend/js/helpers/config.js.example
user: login: greptile-apps[bot]
body: syntax: Typo in clean_name: 'Sytem' should be 'System'

```suggestion
        "clean_name": "Total System Disk Reads",
```
created_at: 2025-05-15T09:57:14Z
author_association: CONTRIBUTOR
diff_hunk: @@ -0,0 +1,45 @@
+from lib import utils
+
+class DiskIoParseMixin:
+    def _parse_metrics(self, df):
+        df = super()._parse_metrics(df)
+
+        df = df.sort_values(by=['detail_name', 'time'], ascending=True)
+
+        df['written_bytes_intervals'] = df.groupby(['detail_name'])['written_bytes'].diff()
+        df['written_bytes_intervals'] = df.groupby('detail_name')['written_bytes_intervals'].transform(utils.df_fill_mean) # fill first NaN value resulted from diff()
+
+        df['read_bytes_intervals'] = df.groupby(['detail_name'])['read_bytes'].diff()
+        df['read_bytes_intervals'] = df.groupby('detail_name')['read_bytes_intervals'].transform(utils.df_fill_mean) # fill first NaN value resulted from diff()
+
+        # we checked at ingest if it contains NA values. So NA can only occur if group diff resulted in only one value.
+        # Since one value is useless for us we drop the row
+        df.dropna(inplace=True)
+
+        if (df['read_bytes_intervals'] < 0).any():
+            raise ValueError('DiskIoCgroupContainerProvider data column read_bytes_intervals had negative values.')
+
+        if (df['written_bytes_intervals'] < 0).any():
+            raise ValueError('DiskIoCgroupContainerProvider data column written_bytes_intervals had negative values.')
path: metric_providers/disk/io/disk_io_parse.py
user: login: greptile-apps[bot]
body: logic: Error messages hardcode 'DiskIoCgroupContainerProvider' but this is a mixin that could be used by other providers. Should use self.__class__.__name__ instead.

```suggestion
        if (df['read_bytes_intervals'] < 0).any():
            raise ValueError(f'{self.__class__.__name__} data column read_bytes_intervals had negative values.')

        if (df['written_bytes_intervals'] < 0).any():
            raise ValueError(f'{self.__class__.__name__} data column written_bytes_intervals had negative values.')
```
created_at: 2025-05-15T09:57:46Z
author_association: CONTRIBUTOR
diff_hunk: @@ -21,22 +21,21 @@ CREATE TABLE measurement_values (
 );
 
 CREATE INDEX measurement_values_mmid ON measurement_values(measurement_metric_id);
path: migrations/2024_12_24_measurement_table_split.sql
user: login: ArneTR
body: Why are you changing an old migration? It was valid at the time ... please revert
created_at: 2025-05-15T10:00:00Z
author_association: MEMBER
diff_hunk: @@ -236,7 +236,6 @@ CREATE TABLE measurement_values (
 );
 
 CREATE INDEX measurement_values_mmid ON measurement_values(measurement_metric_id);
-CREATE UNIQUE INDEX measurement_values_unique ON measurement_values(measurement_metric_id, time);
path: docker/structure.sql
user: login: ArneTR
body: Why does this index get removed? The metrics are different in their metric name and thus get a different measurement_metric_id. it should still be OK?
created_at: 2025-05-15T10:01:17Z
author_association: MEMBER
diff_hunk: @@ -140,12 +140,30 @@ def build_and_store_phase_stats(run_id, sci=None):
                 if metric in ('cpu_utilization_cgroup_container', 'cpu_utilization_cgroup_system', ):
                     cpu_utilization_containers[detail_name] = avg_value
 
-            elif metric in ['network_io_cgroup_system', 'disk_io_cgroup_system', 'network_io_cgroup_container', 'network_io_procfs_system', 'disk_io_procfs_system', 'disk_io_cgroup_container', 'disk_io_bytesread_powermetrics_vm', 'disk_io_byteswritten_powermetrics_vm']:
+            elif metric in ['network_io_cgroup_system',
+                            'network_io_cgroup_container',
+                            'network_io_procfs_system',
+                            'disk_io_procfs_system',
+                            'disk_io_cgroup_container',
+                            'disk_io_bytesread_powermetrics_vm',
+                            'disk_io_byteswritten_powermetrics_vm',
+                            'disk_io_read_cgroup_container',
path: lib/phase_stats.py
user: login: ArneTR
body: one is called bytesread and the other read? Can we only use read? But I understand if you do not want to mix refactorng and feature
created_at: 2025-05-15T10:03:40Z
author_association: MEMBER
diff_hunk: @@ -21,22 +21,21 @@ CREATE TABLE measurement_values (
 );
 
 CREATE INDEX measurement_values_mmid ON measurement_values(measurement_metric_id);
path: migrations/2024_12_24_measurement_table_split.sql
user: login: ribalba
body: Good catch.
created_at: 2025-05-16T07:40:40Z
author_association: MEMBER
diff_hunk: @@ -140,12 +140,30 @@ def build_and_store_phase_stats(run_id, sci=None):
                 if metric in ('cpu_utilization_cgroup_container', 'cpu_utilization_cgroup_system', ):
                     cpu_utilization_containers[detail_name] = avg_value
 
-            elif metric in ['network_io_cgroup_system', 'disk_io_cgroup_system', 'network_io_cgroup_container', 'network_io_procfs_system', 'disk_io_procfs_system', 'disk_io_cgroup_container', 'disk_io_bytesread_powermetrics_vm', 'disk_io_byteswritten_powermetrics_vm']:
+            elif metric in ['network_io_cgroup_system',
+                            'network_io_cgroup_container',
+                            'network_io_procfs_system',
+                            'disk_io_procfs_system',
+                            'disk_io_cgroup_container',
+                            'disk_io_bytesread_powermetrics_vm',
+                            'disk_io_byteswritten_powermetrics_vm',
+                            'disk_io_read_cgroup_container',
path: lib/phase_stats.py
user: login: ribalba
body: I was thinking of changing the powermetrics provider to the new system as a next step and we never add the unit in the name. 
created_at: 2025-05-16T07:41:41Z
author_association: MEMBER
diff_hunk: @@ -0,0 +1,45 @@
+from lib import utils
+
+class DiskIoParseMixin:
+    def _parse_metrics(self, df):
+        df = super()._parse_metrics(df)
+
+        df = df.sort_values(by=['detail_name', 'time'], ascending=True)
+
+        df['written_bytes_intervals'] = df.groupby(['detail_name'])['written_bytes'].diff()
+        df['written_bytes_intervals'] = df.groupby('detail_name')['written_bytes_intervals'].transform(utils.df_fill_mean) # fill first NaN value resulted from diff()
+
+        df['read_bytes_intervals'] = df.groupby(['detail_name'])['read_bytes'].diff()
+        df['read_bytes_intervals'] = df.groupby('detail_name')['read_bytes_intervals'].transform(utils.df_fill_mean) # fill first NaN value resulted from diff()
+
+        # we checked at ingest if it contains NA values. So NA can only occur if group diff resulted in only one value.
+        # Since one value is useless for us we drop the row
+        df.dropna(inplace=True)
+
+        if (df['read_bytes_intervals'] < 0).any():
+            raise ValueError('DiskIoCgroupContainerProvider data column read_bytes_intervals had negative values.')
+
+        if (df['written_bytes_intervals'] < 0).any():
+            raise ValueError('DiskIoCgroupContainerProvider data column written_bytes_intervals had negative values.')
path: metric_providers/disk/io/disk_io_parse.py
user: login: ribalba
body: Not relevant as the Mixin is fixed to the Disk providers
created_at: 2025-05-16T07:48:51Z
author_association: MEMBER
diff_hunk: @@ -140,12 +140,30 @@ def build_and_store_phase_stats(run_id, sci=None):
                 if metric in ('cpu_utilization_cgroup_container', 'cpu_utilization_cgroup_system', ):
                     cpu_utilization_containers[detail_name] = avg_value
 
-            elif metric in ['network_io_cgroup_system', 'disk_io_cgroup_system', 'network_io_cgroup_container', 'network_io_procfs_system', 'disk_io_procfs_system', 'disk_io_cgroup_container', 'disk_io_bytesread_powermetrics_vm', 'disk_io_byteswritten_powermetrics_vm']:
+            elif metric in ['network_io_cgroup_system',
+                            'network_io_cgroup_container',
+                            'network_io_procfs_system',
+                            'disk_io_procfs_system',
+                            'disk_io_cgroup_container',
+                            'disk_io_bytesread_powermetrics_vm',
+                            'disk_io_byteswritten_powermetrics_vm',
+                            'disk_io_read_cgroup_container',
path: lib/phase_stats.py
user: login: ArneTR
body: Ok, please make a todo
created_at: 2025-05-16T09:08:20Z
author_association: MEMBER
diff_hunk: @@ -0,0 +1,45 @@
+from lib import utils
+
+class DiskIoParseMixin:
+    def _parse_metrics(self, df):
+        df = super()._parse_metrics(df)
+
+        df = df.sort_values(by=['detail_name', 'time'], ascending=True)
+
+        df['written_bytes_intervals'] = df.groupby(['detail_name'])['written_bytes'].diff()
+        df['written_bytes_intervals'] = df.groupby('detail_name')['written_bytes_intervals'].transform(utils.df_fill_mean) # fill first NaN value resulted from diff()
+
+        df['read_bytes_intervals'] = df.groupby(['detail_name'])['read_bytes'].diff()
+        df['read_bytes_intervals'] = df.groupby('detail_name')['read_bytes_intervals'].transform(utils.df_fill_mean) # fill first NaN value resulted from diff()
+
+        # we checked at ingest if it contains NA values. So NA can only occur if group diff resulted in only one value.
+        # Since one value is useless for us we drop the row
+        df.dropna(inplace=True)
+
+        if (df['read_bytes_intervals'] < 0).any():
+            raise ValueError('DiskIoCgroupContainerProvider data column read_bytes_intervals had negative values.')
+
+        if (df['written_bytes_intervals'] < 0).any():
+            raise ValueError('DiskIoCgroupContainerProvider data column written_bytes_intervals had negative values.')
path: metric_providers/disk/io/disk_io_parse.py
user: login: ArneTR
body: It is still used by DiskIoCgroupSystemProvider and DiskIoProcfsSystemProvider, so to distinguish WHERE it happened between these two would be nice.

I will add this
created_at: 2025-05-16T09:09:12Z
author_association: MEMBER
diff_hunk: @@ -1484,7 +1484,11 @@ def stop_metric_providers(self):
                 errors.append(f"{metric_provider.__class__.__name__} returned error message: {str(exc)}")
                 continue
 
-            metric_importer.import_measurements(df, metric_provider._metric_name, self._run_id)
+            if isinstance(df, list):
+                for i, dfi in enumerate(df):
+                    metric_importer.import_measurements(dfi, metric_provider._sub_metrics_name[i], self._run_id)
+            else:
+                metric_importer.import_measurements(df, metric_provider._metric_name, self._run_id)
path: lib/scenario_runner.py
user: login: greptile-apps[bot]
body: logic: Consider adding error handling for cases where `_sub_metrics_name` length doesn't match the number of DataFrames in the list
created_at: 2025-05-18T14:53:00Z
author_association: CONTRIBUTOR
diff_hunk: @@ -168,13 +170,14 @@ def read_metrics(self): # should not be overriden
         self._check_resolution_underflow(df)
 
         df = self._parse_metrics(df)
-        df = self._add_auxiliary_fields(df)
 
-        df = self._add_and_validate_sampling_rate_and_jitter(df)
+        def process_df(df):
+            df = self._add_auxiliary_fields(df)
+            df = self._add_and_validate_sampling_rate_and_jitter(df)
+            self._check_empty(df)
+            return df
 
-        self._check_empty(df) # we do another check after transformations, as this could have resulted in zero rows
-
-        return df
+        return process_df(df) if not isinstance(df, list) else [process_df(dfi) for dfi in df]
path: metric_providers/base.py
user: login: greptile-apps[bot]
body: logic: No type checking on df elements when df is a list. Could crash if list contains non-DataFrame objects.

```suggestion
        if isinstance(df, list):
            if not all(isinstance(dfi, pandas.DataFrame) for dfi in df):
                raise TypeError(f"All elements in list must be DataFrames for {self._metric_name}")
            return [process_df(dfi) for dfi in df]
        return process_df(df)
```
created_at: 2025-05-18T14:53:57Z
author_association: CONTRIBUTOR
diff_hunk: @@ -15,33 +15,21 @@ def __init__(self, sampling_rate, skip_check=False):
             skip_check=skip_check,
         )
 
-    def _parse_metrics(self, df):
-        df['detail_name'] = df['device']
-
-        df = df.sort_values(by=['device', 'time'], ascending=True)
-
-        df['read_sectors_intervals'] = df.groupby(['device'])['read_sectors'].diff()
-        df['read_sectors_intervals'] = df.groupby('device')['read_sectors_intervals'].transform(utils.df_fill_mean) # fill first NaN value resulted from diff()
-
-        df['written_sectors_intervals'] = df.groupby(['device'])['written_sectors'].diff()
-        df['written_sectors_intervals'] = df.groupby('device')['written_sectors_intervals'].transform(utils.df_fill_mean) # fill first NaN value resulted from diff()
+        self._sub_metrics_name = ['disk_io_read_procfs_system', 'disk_io_write_procfs_system']
 
-        # we checked at ingest if it contains NA values. So NA can only occur if group diff resulted in only one value.
-        # Since one value is useless for us we drop the row
-        df.dropna(inplace=True)
-
-        if (df['written_sectors_intervals'] < 0).any():
-            raise ValueError('DiskIoProcfsSystemProvider data column written_sectors_intervals had negative values.')
+    def _parse_metrics(self, df):
+        df = super()._parse_metrics(df)
+        return self._parse_metrics_splitup_helper(df)
path: metric_providers/disk/io/procfs/system/provider.py
user: login: greptile-apps[bot]
body: logic: This method is duplicated with the one below and will be overridden. Remove this duplicate definition.

```suggestion

```
created_at: 2025-05-18T14:54:03Z
author_association: CONTRIBUTOR
diff_hunk: @@ -0,0 +1,44 @@
+from lib import utils
+
+class DiskIoParseMixin:
+    def _parse_metrics_splitup_helper(self, df):
+
+        df = df.sort_values(by=['detail_name', 'time'], ascending=True)
+
+        df['written_bytes_intervals'] = df.groupby(['detail_name'])['written_bytes'].diff()
+        df['written_bytes_intervals'] = df.groupby('detail_name')['written_bytes_intervals'].transform(utils.df_fill_mean) # fill first NaN value resulted from diff()
+
+        df['read_bytes_intervals'] = df.groupby(['detail_name'])['read_bytes'].diff()
+        df['read_bytes_intervals'] = df.groupby('detail_name')['read_bytes_intervals'].transform(utils.df_fill_mean) # fill first NaN value resulted from diff()
+
+        # we checked at ingest if it contains NA values. So NA can only occur if group diff resulted in only one value.
+        # Since one value is useless for us we drop the row
+        df.dropna(inplace=True)
+
+        if (df['read_bytes_intervals'] < 0).any():
+            raise ValueError(f"{self.__class__.__name__} data column read_bytes_intervals had negative values.")
+
+        if (df['written_bytes_intervals'] < 0).any():
+            raise ValueError(f"{self.__class__.__name__} data column written_bytes_intervals had negative values.")
+
+        base_cols = ['time', 'detail_name']
+
+        df_read = (
+            df[base_cols + ["read_bytes_intervals"]]
+            .rename(columns={"read_bytes_intervals": "value"})
+            .copy()
+        )
+        df_read["value"] = df_read["value"].astype(int)
path: metric_providers/disk/io/disk_io_parse.py
user: login: greptile-apps[bot]
body: logic: astype(int) could truncate large values. Should use int64 to handle large disk operations.

```suggestion
        df_read["value"] = df_read["value"].astype('int64')
```
created_at: 2025-05-18T14:54:07Z
author_association: CONTRIBUTOR

PR comments:
user: login: ArneTR
created_at: 2025-05-15T09:56:36Z
body: @greptileai
author_association: MEMBER
user: login: ArneTR
created_at: 2025-05-15T11:21:58Z
body: Also nice to see that you do not write descriptions for PRs anymore, as Greptile does it :)
author_association: MEMBER
user: login: ribalba
created_at: 2025-05-16T07:48:04Z
body: Good catch with the indexes. That was a remnant from when I was trying out different things. Fixed most things. 

Regarding the Mixin. This is functionality that is only for the DiskIO providers. It is the same for both of them but can not be used for the network provider as it is disk specific. How would you share a function between two classes. I was thinking of creating a new class that only has this method and then extend from this. But because of the init I found this to be more confusing. Otherwise I could copy the code. Or I could import the function and then use it as a shared lib. But that also didn't look very "clean"
author_association: MEMBER
user: login: ArneTR
created_at: 2025-05-18T14:51:57Z
body: @ribalba 

- I removed the hack to get the target sample rate in phase_stats. Thanks for the nudge. Is way cleaner now
- I was not sure why you did not bring this functionality to disk_io_procfs_system also. So I altered the mixin a little. Can you give it a look
author_association: MEMBER
user: login: ArneTR
created_at: 2025-05-18T14:52:06Z
body: @greptileai
author_association: MEMBER