Repository: ansible/metrics-utility
Pull request #86: Build report searches through only the sheets the user is requesting.
Author: login: AlexSCorey
Created: 2025-03-10T19:57:34Z | Merged: 2025-03-18T16:13:43Z

[AAP-41408]


## Description
This PR optimizes code so that when a user runs a report for only specific data we are only gathering the reports for the data requested.  This pr should not change any functionality.
## Testing
Current tests should suffice

### Code Quality

- [x] Code is properly linted and formatted.
- [x] All tests (existing and new) pass successfully.
- [x] Tests are added or updated as needed.
- [x] Code includes relevant comments for complex sections.
- [x] Changes are reviewed and approved by at least two team members.
- [x] Documentation is updated where applicable.  
       - If updates are required in the [handbook](https://github.com/ansible/handbook), create a separate PR for the handbook repo.

## Notes for Reviewers

Add any additional context or instructions for reviewers here - for example screenshots if needed.



Diff:
diff --git a/metrics_utility/automation_controller_billing/extract/base.py b/metrics_utility/automation_controller_billing/extract/base.py
new file mode 100644
index 000000000..9900b2828
--- /dev/null
+++ b/metrics_utility/automation_controller_billing/extract/base.py
@@ -0,0 +1,148 @@
+import pandas as pd
+import os
+import tarfile
+
+
+class Base:
+    def process_tarballs(self, path, temp_dir):
+        _safe_extract(path, temp_dir)
+        config = self.load_config(os.path.join(temp_dir, 'config.json'))
+
+        # # TODO: read the csvs in batches
+        # for chunk in pd.read_csv(filename, chunksize=chunksize):
+        # # chunk is a DataFrame. To "process" the rows in the chunk:
+        # for index, row in chunk.iterrows():
+        #     print(row)
+
+        empty_dataframe = pd.DataFrame([{}])
+        needed_data = {
+            'config': config,
+            'job_host_summary': empty_dataframe,
+            'indirect_nodes': empty_dataframe,
+            'main_jobevent': empty_dataframe,
+            'main_host': empty_dataframe,
+        }
+        if self.required_sheets(
+            [
+                'ccsp_summary',
+                'managed_nodes',
+                'indirectly_managed_nodes',
+                'inventory_scope',
+                'usage_by_organizations',
+                'managed_nodes_by_organizations',
+            ]
+        ):
+            needed_data['job_host_summary'] = self.build_data_batch(temp_dir, 'job_host_summary')
+
+        if self.required_sheets(['managed_nodes', 'indirectly_managed_nodes', 'usage_by_organizations']):
+            needed_data['indirect_nodes'] = self.build_data_batch(temp_dir, 'main_indirectmanagednodeaudit')
+
+        if self.required_sheets(['usage_by_collections', 'usage_by_roles', 'usage_by_modules', 'usage_by_organizations']):
+            needed_data['main_jobevent'] = self.build_data_batch(temp_dir, 'main_jobevent')
+
+        if self.required_sheets(
+            [
+                'jobs',
+                'managed_nodes_by_organizations',
+                'managed_nodes',
+                'inventory_scope',
+                'usage_by_collections',
+                'usage_by_modules',
+                'managed_nodes',
+            ]
+        ):
+            needed_data['main_host'] = self.build_data_batch(temp_dir, 'main_host')
+
+        return needed_data
+
+    def build_data_batch(self, temp_dir, file_name):
+        """
+        Builds the report with only the necessary sheets.
+        """
+
+        if os.path.exists(os.path.join(temp_dir, f'{file_name}.csv')):
+            return pd.read_csv(os.path.join(temp_dir, f'{file_name}.csv'))
+        else:
+            return pd.DataFrame([{}])
+
+    def required_sheets(self, sheets_required):
+        """
+        Checks for METRICS_UTILITY_OPTIONAL_CCSP_REPORT_SHEETS values and return those values.
+        If none found, give it the default CCSP report sheet options.
+        Returns a boolean so we know which sheets to provide in the report.
+        """
+        sheet_options = self.extra_params.get('optional_sheets')
+        return bool(set(sheet_options) & set(sheets_required))
+
+
+def _write_member(member_path, file_obj, max_size, total_extracted_size):
+    with open(member_path, 'wb') as out_f:
+        chunk_size = 1024 * 1024  # 1 MB buffer
+        while True:
+            data = file_obj.read(chunk_size)
+            if not data:
+                break
+            total_extracted_size += len(data)
+            if total_extracted_size > max_size:
+                # Stop if we exceed total extraction size
+                raise ValueError('Extraction aborted: Maximum total size exceeded.')
+            out_f.write(data)
+    return total_extracted_size
+
+
+def _safe_extract(tar_path, extract_path, max_files=100, max_size=1024 * 1024 * 1024):
+    """
+    Safely extract a tar archive from 'tar_path' into 'extract_path' with constraints:
+      - Only extract *.json or *.csv files
+      - Skip directories, symbolic links, and hard links
+      - Limit number of extracted files to 'max_files'
+      - Limit total uncompressed size to 'max_size' bytes
+    """
+    extracted_files = 0
+    total_extracted_size = 0
+
+    # Ensure the extraction directory exists
+    os.makedirs(extract_path, exist_ok=True)
+
+    with tarfile.open(tar_path, 'r:*') as tar:
+        for member in tar.getmembers():
+            # 1) Skip directories and links
+            if member.isdir():
+                continue
+            if member.issym() or member.islnk():
+                print(f'Skipping link: {member.name}')
+                continue
+
+            # 2) Only allow .json or .csv
+            if not (member.name.endswith('.json') or member.name.endswith('.csv')):
+                continue
+
+            # 3) Build a fully qualified path for this member
+            #    and ensure it stays within extract_path.
+            member_path = os.path.abspath(os.path.join(extract_path, member.name))
+            extract_path_abs = os.path.abspath(extract_path)
+            if not member_path.startswith(extract_path_abs + os.sep):
+                print(f'Skipping potentially unsafe file (path traversal): {member.name}')
+                continue
+
+            # 4) Limit total files
+            if extracted_files >= max_files:
+                print(f'Reached max file limit of {max_files}.')
+                break
+
+            # 5) Extract file contents manually, in chunks,
+            #    to avoid trusting the tar's metadata size.
+            file_obj = tar.extractfile(member)
+            if file_obj is None:
+                # Could not read the file content for some reason
+                continue
+
+            # Make sure the subdirectory structure exists
+            os.makedirs(os.path.dirname(member_path), exist_ok=True)
+
+            # Write out the file, limiting max size
+            total_extracted_size = _write_member(member_path, file_obj, max_size, total_extracted_size)
+
+            extracted_files += 1
+
+    print(f'Extraction complete. Files extracted: {extracted_files}, Total size: {total_extracted_size} bytes.')
diff --git a/metrics_utility/automation_controller_billing/extract/extractor_common.py b/metrics_utility/automation_controller_billing/extract/extractor_common.py
deleted file mode 100644
index f931e3beb..000000000
--- a/metrics_utility/automation_controller_billing/extract/extractor_common.py
+++ /dev/null
@@ -1,115 +0,0 @@
-import pandas as pd
-import os
-import tarfile
-
-
-def write_member(member_path, file_obj, max_size, total_extracted_size):
-    with open(member_path, 'wb') as out_f:
-        chunk_size = 1024 * 1024  # 1 MB buffer
-        while True:
-            data = file_obj.read(chunk_size)
-            if not data:
-                break
-            total_extracted_size += len(data)
-            if total_extracted_size > max_size:
-                # Stop if we exceed total extraction size
-                raise ValueError("Extraction aborted: Maximum total size exceeded.")
-            out_f.write(data)
-    return total_extracted_size
-
-
-def safe_extract(tar_path, extract_path, max_files=100, max_size=1024*1024*1024):
-    """
-    Safely extract a tar archive from 'tar_path' into 'extract_path' with constraints:
-      - Only extract *.json or *.csv files
-      - Skip directories, symbolic links, and hard links
-      - Limit number of extracted files to 'max_files'
-      - Limit total uncompressed size to 'max_size' bytes
-    """
-    extracted_files = 0
-    total_extracted_size = 0
-
-    # Ensure the extraction directory exists
-    os.makedirs(extract_path, exist_ok=True)
-
-    with tarfile.open(tar_path, 'r:*') as tar:
-        for member in tar.getmembers():
-
-            # 1) Skip directories and links
-            if member.isdir():
-                continue
-            if member.issym() or member.islnk():
-                print(f"Skipping link: {member.name}")
-                continue
-
-            # 2) Only allow .json or .csv
-            if not (member.name.endswith('.json') or member.name.endswith('.csv')):
-                continue
-
-            # 3) Build a fully qualified path for this member
-            #    and ensure it stays within extract_path.
-            member_path = os.path.abspath(os.path.join(extract_path, member.name))
-            extract_path_abs = os.path.abspath(extract_path)
-            if not member_path.startswith(extract_path_abs + os.sep):
-                print(f"Skipping potentially unsafe file (path traversal): {member.name}")
-                continue
-
-            # 4) Limit total files
-            if extracted_files >= max_files:
-                print(f"Reached max file limit of {max_files}.")
-                break
-
-            # 5) Extract file contents manually, in chunks,
-            #    to avoid trusting the tar's metadata size.
-            file_obj = tar.extractfile(member)
-            if file_obj is None:
-                # Could not read the file content for some reason
-                continue
-
-            # Make sure the subdirectory structure exists
-            os.makedirs(os.path.dirname(member_path), exist_ok=True)
-
-            # Write out the file, limiting max size
-            total_extracted_size = write_member(member_path, file_obj, max_size, total_extracted_size)
-
-            extracted_files += 1
-
-    print(f"Extraction complete. Files extracted: {extracted_files}, "
-          f"Total size: {total_extracted_size} bytes.")
-
-def process_tarballs(self, path, temp_dir):
-    safe_extract(path, temp_dir)
-    config = self.load_config(os.path.join(temp_dir, 'config.json'))
-
-    # # TODO: read the csvs in batches
-    # for chunk in pd.read_csv(filename, chunksize=chunksize):
-    # # chunk is a DataFrame. To "process" the rows in the chunk:
-    # for index, row in chunk.iterrows():
-    #     print(row)
-
-    if os.path.exists(os.path.join(temp_dir, 'job_host_summary.csv')):
-        job_host_summary = pd.read_csv(os.path.join(temp_dir, 'job_host_summary.csv'))
-    else:
-        job_host_summary = pd.DataFrame([{}])
-
-    if os.path.exists(os.path.join(temp_dir, 'main_indirectmanagednodeaudit.csv')):
-        indirect_nodes = pd.read_csv(os.path.join(temp_dir, 'main_indirectmanagednodeaudit.csv'))
-    else:
-        indirect_nodes = pd.DataFrame([{}])
-
-    if os.path.exists(os.path.join(temp_dir, 'main_jobevent.csv')):
-        main_jobevent = pd.read_csv(os.path.join(temp_dir, 'main_jobevent.csv'))
-    else:
-        main_jobevent = pd.DataFrame([{}])
-
-    if os.path.exists(os.path.join(temp_dir, 'main_host.csv')):
-        main_host = pd.read_csv(os.path.join(temp_dir, 'main_host.csv'))
-    else:
-        main_host = pd.DataFrame([{}])
-
-
-    return {'main_jobevent': main_jobevent,
-            'main_host': main_host,
-            'job_host_summary': job_host_summary,
-            'indirect_nodes' : indirect_nodes,
-            'config': config}
diff --git a/metrics_utility/automation_controller_billing/extract/extractor_directory.py b/metrics_utility/automation_controller_billing/extract/extractor_directory.py
index 43fbb8d20..961940933 100644
--- a/metrics_utility/automation_controller_billing/extract/extractor_directory.py
+++ b/metrics_utility/automation_controller_billing/extract/extractor_directory.py
@@ -4,39 +4,39 @@
 import os
 import tempfile
 import pandas as pd
+from metrics_utility.automation_controller_billing.extract.base import Base
 
-from metrics_utility.automation_controller_billing.extract.extractor_common import process_tarballs
 
-class ExtractorDirectory:
-    LOG_PREFIX = "[ExtractorDirectory]"
+class ExtractorDirectory(Base):
+    LOG_PREFIX = '[ExtractorDirectory]'
 
     def __init__(self, extra_params, logger=logging.getLogger(__name__)):
         super().__init__()
 
-        self.extension = "parquet"
-        self.path = extra_params["ship_path"]
+        self.extension = 'parquet'
+        self.path = extra_params['ship_path']
         self.extra_params = extra_params
 
         self.logger = logger
 
     def _get_path_prefix(self, date):
-        path_prefix = f"{self.path}/data"
+        path_prefix = f'{self.path}/data'
 
-        year = date.strftime("%Y")
-        month = date.strftime("%m")
-        day = date.strftime("%d")
+        year = date.strftime('%Y')
+        month = date.strftime('%m')
+        day = date.strftime('%d')
 
-        path = f"{path_prefix}/{year}/{month}/{day}"
+        path = f'{path_prefix}/{year}/{month}/{day}'
 
         return path
 
     def get_report_path(self, date):
-        path_prefix = f"{self.path}/reports"
+        path_prefix = f'{self.path}/reports'
 
-        year = date.strftime("%Y")
-        month = date.strftime("%m")
+        year = date.strftime('%Y')
+        month = date.strftime('%m')
 
-        path = f"{path_prefix}/{year}/{month}"
+        path = f'{path_prefix}/{year}/{month}'
 
         return path
 
@@ -44,7 +44,7 @@ def iter_batches(self, date, columns=None, batch_size=None):
         if batch_size is None:
             batch_size = self.batch_size()
         # Read parquet in memory in batches
-        self.logger.info(f"{self.LOG_PREFIX} Processing {date}")
+        self.logger.info(f'{self.LOG_PREFIX} Processing {date}')
         paths = self.fetch_partition_paths(date)
 
         if batch_size is None:
@@ -54,12 +54,12 @@ def iter_batches(self, date, columns=None, batch_size=None):
             if not path.endswith('.tar.gz'):
                 continue
 
-            with tempfile.TemporaryDirectory(prefix="automation_controller_billing_data_") as temp_dir:
+            with tempfile.TemporaryDirectory(prefix='automation_controller_billing_data_') as temp_dir:
                 try:
-                    yield process_tarballs(self, path, temp_dir)
+                    yield self.process_tarballs(path, temp_dir)
 
                 except Exception as e:
-                    self.logger.exception(f"{self.LOG_PREFIX} ERROR: Extracting {path} failed with {e}")
+                    self.logger.exception(f'{self.LOG_PREFIX} ERROR: Extracting {path} failed with {e}')
 
     def load_config(self, file_path):
         try:
@@ -67,7 +67,7 @@ def load_config(self, file_path):
                 config_data = json.loads(f.read())
             return config_data
         except FileNotFoundError:
-            self.logger.warn(f"{self.LOG_PREFIX} missing required file under path: {self.path} and date: {self.date}")
+            self.logger.warn(f'{self.LOG_PREFIX} missing required file under path: {self.path} and date: {self.date}')
             # raise MissingRequiredFile(self.filename) from e
 
     def fetch_partition_paths(self, date):
@@ -80,10 +80,9 @@ def fetch_partition_paths(self, date):
 
         return paths
 
-    def mapping(self, normalized_table, id_column="id", value_column="value"):
+    def mapping(self, normalized_table, id_column='id', value_column='value'):
         try:
-            mapping = self.read_parquet_file(
-                f"data/parquet/{self.tenant_id}/{normalized_table}/{normalized_table}.parquet")
+            mapping = self.read_parquet_file(f'data/parquet/{self.tenant_id}/{normalized_table}/{normalized_table}.parquet')
             mapping.set_index(id_column, inplace=True)
             return mapping[value_column].astype(str).to_dict()
         except Exception:
@@ -100,14 +99,12 @@ def read_parquet_file(self, obj, columns=None, raise_exception=False, silence_ex
                 return pd.read_parquet(buffer)
         except Exception as e:
             if not silence_exception:
-                self.logger.error(f"ERROR: {obj} failed with {e}")
+                self.logger.error(f'ERROR: {obj} failed with {e}')
             if raise_exception:
                 raise e
 
-    def read_parquet_files(self, objs, columns=None,
-                           raise_exception=False, silence_exception=False):
-        dfs = [self.read_parquet_file(obj, columns, raise_exception, silence_exception)
-               for obj in objs if obj is not None]
+    def read_parquet_files(self, objs, columns=None, raise_exception=False, silence_exception=False):
+        dfs = [self.read_parquet_file(obj, columns, raise_exception, silence_exception) for obj in objs if obj is not None]
         dfs = [df for df in dfs if df is not None]
         if len(dfs) > 0:
             return pd.concat(dfs, ignore_index=True)
diff --git a/metrics_utility/automation_controller_billing/extract/extractor_s3.py b/metrics_utility/automation_controller_billing/extract/extractor_s3.py
index a07046215..55d4ee514 100644
--- a/metrics_utility/automation_controller_billing/extract/extractor_s3.py
+++ b/metrics_utility/automation_controller_billing/extract/extractor_s3.py
@@ -3,43 +3,43 @@
 import os
 import tempfile
 
-from metrics_utility.automation_controller_billing.base.s3_handler import \
-    S3Handler
+from metrics_utility.automation_controller_billing.base.s3_handler import S3Handler
+from metrics_utility.automation_controller_billing.extract.base import Base
 
-from metrics_utility.automation_controller_billing.extract.extractor_common import process_tarballs
 
-class ExtractorS3:
-    LOG_PREFIX = "[ExtractorS3]"
+class ExtractorS3(Base):
+    LOG_PREFIX = '[ExtractorS3]'
 
     def __init__(self, extra_params, logger=logging.getLogger(__name__)):
         super().__init__()
 
-        self.extension = "parquet"
-        self.path = extra_params["ship_path"]
+        self.extension = 'parquet'
+        self.path = extra_params['ship_path']
         self.extra_params = extra_params
-
         self.logger = logger
 
         self.s3_handler = S3Handler(params=self.extra_params)
 
-    def _get_path_prefix(self, date):
-        path_prefix = f"{self.path}/data"
+    def _create_date_string(self, date):
+        year = date.strftime('%Y')
+        month = date.strftime('%m')
+        day = date.strftime('%d')
 
-        year = date.strftime("%Y")
-        month = date.strftime("%m")
-        day = date.strftime("%d")
+        return {year, month, day}
 
-        path = f"{path_prefix}/{year}/{month}/{day}"
+    def _get_path_prefix(self, date):
+        data_path_prefix = f'{self.path}/data'
+
+        year, month, day = self._create_date_string(date)
+        path = f'{data_path_prefix}/{year}/{month}/{day}'
 
         return path
 
     def get_report_path(self, date):
-        path_prefix = f"{self.path}/reports"
-
-        year = date.strftime("%Y")
-        month = date.strftime("%m")
+        report_path_prefix = f'{self.path}/reports'
 
-        path = f"{path_prefix}/{year}/{month}"
+        year, month = self._create_date_string(date)
+        path = f'{report_path_prefix}/{year}/{month}'
 
         return path
 
@@ -47,22 +47,22 @@ def iter_batches(self, date, columns=None, batch_size=None):
         if batch_size is None:
             batch_size = self.batch_size()
         # Read parquet in memory in batches
-        self.logger.info(f"{self.LOG_PREFIX} Processing {date}")
+        self.logger.info(f'{self.LOG_PREFIX} Processing {date}')
         s3_paths = self.fetch_partition_paths(date)
 
         if batch_size is None:
             batch_size = self.batch_size()
 
         for s3_path in s3_paths:
-            with tempfile.TemporaryDirectory(prefix="automation_controller_billing_data_") as temp_dir:
+            with tempfile.TemporaryDirectory(prefix='automation_controller_billing_data_') as temp_dir:
                 try:
                     local_path = os.path.join(temp_dir, 'source_tarball')
                     self.s3_handler.download_file(s3_path, local_path)
 
-                    yield process_tarballs(self, s3_path, temp_dir)
+                    yield self.process_tarballs(self, s3_path, temp_dir)
 
                 except Exception as e:
-                    self.logger.exception(f"{self.LOG_PREFIX} ERROR: Extracting {s3_path} failed with {e}")
+                    self.logger.exception(f'{self.LOG_PREFIX} ERROR: Extracting {s3_path} failed with {e}')
 
     def load_config(self, file_path):
         try:
@@ -70,7 +70,7 @@ def load_config(self, file_path):
                 config_data = json.loads(f.read())
             return config_data
         except FileNotFoundError:
-            self.logger.warn(f"{self.LOG_PREFIX} missing required file under path: {self.path} and date: {self.date}")
+            self.logger.warn(f'{self.LOG_PREFIX} missing required file under path: {self.path} and date: {self.date}')
             # raise MissingRequiredFile(self.filename) from e
 
     def fetch_partition_paths(self, date):
diff --git a/metrics_utility/automation_controller_billing/report/base.py b/metrics_utility/automation_controller_billing/report/base.py
index 4429db26b..7dc7335c1 100644
--- a/metrics_utility/automation_controller_billing/report/base.py
+++ b/metrics_utility/automation_controller_billing/report/base.py
@@ -1,7 +1,6 @@
 ######################################
 # Code for building the spreadsheet
 ######################################
-import os
 import json
 
 from openpyxl.styles import Font
@@ -11,21 +10,23 @@
 
 
 class Base:
-    BLACK_COLOR_HEX = "00000000"
-    WHITE_COLOR_HEX = "00FFFFFF"
-    BLUE_COLOR_HEX = "000000FF"
-    RED_COLOR_HEX = "FF0000"
-    LIGHT_BLUE_COLOR_HEX = "d4eaf3"
-    GREEN_COLOR_HEX = "92d050"
-    FONT = "Arial"
+    BLACK_COLOR_HEX = '00000000'
+    WHITE_COLOR_HEX = '00FFFFFF'
+    BLUE_COLOR_HEX = '000000FF'
+    RED_COLOR_HEX = 'FF0000'
+    LIGHT_BLUE_COLOR_HEX = 'd4eaf3'
+    GREEN_COLOR_HEX = '92d050'
+    FONT = 'Arial'
     PRICE_FORMAT = '$#,##0.00'
+    HOST_NAME = 'Host name'
+    JOB_RUNS ='Job runs'
+    NUM_OF_TASKS_OR_RUNS = 'Number of task\nruns'
+    HOST_RUNS_UNIQUE = 'Unique managed nodes\nautomated'
+    HOST_RUNS = 'Non-unique managed\nnodes automated'
+    DURATION = 'Duration of task\nruns [seconds]'
 
-    @staticmethod
-    def optional_report_sheets():
-        return os.environ.get(
-            'METRICS_UTILITY_OPTIONAL_CCSP_REPORT_SHEETS',
-            'ccsp_summary,managed_nodes,usage_by_organizations,usage_by_collections,usage_by_roles,'\
-            'usage_by_modules').split(",")
+    def optional_report_sheets(self):
+        return self.extra_params.get('optional_sheets')
 
     def convert_cell(self, cell):
         # If the cell is a dictionary, convert each set value to a sorted list, then dump as a JSON string.
@@ -47,47 +48,38 @@ def _fix_event_host_names(self, mapping_dataframe, destination_dataframe):
             return None
 
         def concatenate_columns_mapping(row):
-            return f"{row['original_host_name']}__{str(row['install_uuid'])}__{str(row['job_remote_id'])}"
+            return f'{row["original_host_name"]}__{str(row["install_uuid"])}__{str(row["job_remote_id"])}'
 
         def concatenate_columns_destination(row):
-            return f"{row['host_name']}__{str(row['install_uuid'])}__{str(row['job_remote_id'])}"
+            return f'{row["host_name"]}__{str(row["install_uuid"])}__{str(row["job_remote_id"])}'
 
         # Apply the function to each row of the DataFrame
         mapping_dataframe['host_composite_id'] = mapping_dataframe.apply(concatenate_columns_mapping, axis=1)
-        mapping_dataframe = mapping_dataframe.set_index("host_composite_id")
-        mapping_dataframe = mapping_dataframe["host_name"].astype(str).to_dict()
+        mapping_dataframe = mapping_dataframe.set_index('host_composite_id')
+        mapping_dataframe = mapping_dataframe['host_name'].astype(str).to_dict()
 
         def apply_mapping(row):
-            return mapping_dataframe.get(f"{row['host_name']}__{str(row['install_uuid'])}__{row['job_remote_id']}", row['host_name'])
+            return mapping_dataframe.get(f'{row["host_name"]}__{str(row["install_uuid"])}__{row["job_remote_id"]}', row['host_name'])
 
         destination_dataframe['host_name'] = destination_dataframe.apply(apply_mapping, axis=1)
         destination_dataframe['host_composite_id'] = destination_dataframe.apply(concatenate_columns_destination, axis=1)
 
         return destination_dataframe
 
-
     def _build_data_section_usage_by_node_with_org_details(self, current_row, ws, dataframe, mode=None, managed_node_type=None):
         for key, value in self.config['data_column_widths'].items():
             ws.column_dimensions[get_column_letter(key)].width = value
 
-        header_font = Font(name=self.FONT,
-                           size=10,
-                           color=self.BLACK_COLOR_HEX,
-                           bold=True)
-        value_font = Font(name=self.FONT,
-                          size=10,
-                          color=self.BLACK_COLOR_HEX)
+        header_font = Font(name=self.FONT, size=10, color=self.BLACK_COLOR_HEX, bold=True)
+        value_font = Font(name=self.FONT, size=10, color=self.BLACK_COLOR_HEX)
 
         # Rename the columns based on the template
-        ccsp_report_dataframe = (
-            dataframe.groupby('host_name', dropna=False)
-            .agg(
-                organizations=('organization_name', 'nunique'),
-                host_runs=('host_name', 'count'),
-                task_runs=('task_runs', 'sum'),
-                first_automation=('first_automation', 'min'),
-                last_automation=('last_automation', 'max')
-            )
+        ccsp_report_dataframe = dataframe.groupby('host_name', dropna=False).agg(
+            organizations=('organization_name', 'nunique'),
+            host_runs=('host_name', 'count'),
+            task_runs=('task_runs', 'sum'),
+            first_automation=('first_automation', 'min'),
+            last_automation=('last_automation', 'max'),
         )
         ccsp_report_dataframe = ccsp_report_dataframe.reset_index()
         columns = [
@@ -98,20 +90,18 @@ def _build_data_section_usage_by_node_with_org_details(self, current_row, ws, da
             'first_automation',
             'last_automation',
         ]
-        if mode == "by_organization":
+        if mode == 'by_organization':
             # Filter some columns out based on mode
             columns = [col for col in columns if col not in ['organizations']]
 
-        ccsp_report_dataframe = ccsp_report_dataframe.reindex(
-            columns=columns
-        )
+        ccsp_report_dataframe = ccsp_report_dataframe.reindex(columns=columns)
 
         # Create dataframe with hostname and orgs as columns, having last automation for each host
         pivoted_dataframe = dataframe.pivot_table(
             index='host_name',
             columns='organization_name',
             values='last_automation',
-            aggfunc='max'  # You can use 'max', 'min', 'mean', etc., depending on your needs
+            aggfunc='max',  # You can use 'max', 'min', 'mean', etc., depending on your needs
         )
 
         # Set index on host_name for join
@@ -122,17 +112,15 @@ def _build_data_section_usage_by_node_with_org_details(self, current_row, ws, da
         ccsp_report_dataframe = ccsp_report_dataframe.reset_index()
 
         labels = {
-            "host_name": "Host name",
-            "organizations": "Automated by\norganizations",
-            "host_runs": "Job runs",  # Job runs is the same as host_runs, Non-unique managed nodes automated
-            "task_runs": "Number of task\nruns",
-            'first_automation': "First\nautomation",
-            'last_automation': "Last\nautomation",
+            'host_name': self.HOST_NAME,
+            'organizations': 'Automated by\norganizations',
+            'host_runs': self.JOB_RUNS,  # Job runs is the same as host_runs, Non-unique managed nodes automated
+            'task_runs': self.NUM_OF_TASKS_OR_RUNS,
+            'first_automation': 'First\nautomation',
+            'last_automation': 'Last\nautomation',
         }
-        labels = {k:v for k, v in labels.items() if k in columns}
-        ccsp_report_dataframe = ccsp_report_dataframe.rename(
-            columns=labels
-        )
+        labels = {k: v for k, v in labels.items() if k in columns}
+        ccsp_report_dataframe = ccsp_report_dataframe.rename(columns=labels)
 
         row_counter = 0
         rows = dataframe_to_rows(ccsp_report_dataframe, index=False)
@@ -154,57 +142,39 @@ def _build_data_section_usage_by_node_with_org_details(self, current_row, ws, da
 
         return current_row + row_counter
 
-
     def _build_data_section_usage_by_job(self, current_row, ws, dataframe):
         for key, value in self.config['data_column_widths'].items():
             ws.column_dimensions[get_column_letter(key)].width = value
 
-        header_font = Font(name=self.FONT,
-                           size=10,
-                           color=self.BLACK_COLOR_HEX,
-                           bold=True)
-        value_font = Font(name=self.FONT,
-                          size=10,
-                          color=self.BLACK_COLOR_HEX)
+        header_font = Font(name=self.FONT, size=10, color=self.BLACK_COLOR_HEX, bold=True)
+        value_font = Font(name=self.FONT, size=10, color=self.BLACK_COLOR_HEX)
 
         dataframe['job_remote_id_install_uuid'] = list(zip(dataframe['job_remote_id'], dataframe['install_uuid']))
 
         # Rename the columns based on the template
-        ccsp_report_dataframe = (
-            dataframe.groupby(['organization_name', 'job_template_name'], dropna=False)
-            .agg(
-                job_runs=('job_remote_id_install_uuid', 'nunique'),
-                host_runs_unique=('host_name', 'nunique'),
-                host_runs=('host_name', 'count'),
-                task_runs=('task_runs', 'sum'),
-                first_run=('job_created', 'min'),
-                last_run=('job_created', 'max'),
-            )
+        ccsp_report_dataframe = dataframe.groupby(['organization_name', 'job_template_name'], dropna=False).agg(
+            job_runs=('job_remote_id_install_uuid', 'nunique'),
+            host_runs_unique=('host_name', 'nunique'),
+            host_runs=('host_name', 'count'),
+            task_runs=('task_runs', 'sum'),
+            first_run=('job_created', 'min'),
+            last_run=('job_created', 'max'),
         )
         ccsp_report_dataframe = ccsp_report_dataframe.reset_index()
         ccsp_report_dataframe = ccsp_report_dataframe.reindex(
-            columns=[
-                'job_template_name',
-                'organization_name',
-                'job_runs',
-                'host_runs_unique',
-                'host_runs',
-                'task_runs',
-                'first_run',
-                'last_run'
-            ]
+            columns=['job_template_name', 'organization_name', 'job_runs', 'host_runs_unique', 'host_runs', 'task_runs', 'first_run', 'last_run']
         )
 
         ccsp_report_dataframe = ccsp_report_dataframe.rename(
             columns={
-                "job_template_name": "Job template\nname",
-                "organization_name": "Organization\nname",
-                "job_runs": "Job runs",
-                "host_runs_unique": "Unique managed nodes\nautomated",
-                "host_runs": "Non-unique managed\nnodes automated",
-                "task_runs": "Number of task\nruns",
-                "first_run": "First\nrun",
-                "last_run": "Last\nrun",
+                'job_template_name': 'Job template\nname',
+                'organization_name': 'Organization\nname',
+                'job_runs': self.JOB_RUNS,
+                'host_runs_unique': self.HOST_RUNS_UNIQUE,
+                'host_runs': self.HOST_RUNS,
+                'task_runs': self.NUM_OF_TASKS_OR_RUNS,
+                'first_run': 'First\nrun',
+                'last_run': 'Last\nrun',
             }
         )
 
@@ -232,13 +202,8 @@ def _build_data_section_scope(self, current_row, ws, dataframe, mode=None):
         for key, value in self.config['data_column_widths'].items():
             ws.column_dimensions[get_column_letter(key)].width = value
 
-        header_font = Font(name=self.FONT,
-                           size=10,
-                           color=self.BLACK_COLOR_HEX,
-                           bold=True)
-        value_font = Font(name=self.FONT,
-                          size=10,
-                          color=self.BLACK_COLOR_HEX)
+        header_font = Font(name=self.FONT, size=10, color=self.BLACK_COLOR_HEX, bold=True)
+        value_font = Font(name=self.FONT, size=10, color=self.BLACK_COLOR_HEX)
 
         ccsp_report_dataframe = dataframe
 
@@ -259,17 +224,15 @@ def _build_data_section_scope(self, current_row, ws, dataframe, mode=None):
         ]
 
         labels = {
-            "host_name": "Host name",
-            "last_automation": "Last\nAutomation",
-            "organizations": "Organizations",
-            "inventories": "Inventories",  # Job runs is the same as host_runs, Non-unique managed nodes automated
-            "canonical_facts": "Canonical Facts",
-            'facts': "Facts",
+            'host_name': self.HOST_NAME,
+            'last_automation': 'Last\nAutomation',
+            'organizations': 'Organizations',
+            'inventories': 'Inventories',  # Job runs is the same as host_runs, Non-unique managed nodes automated
+            'canonical_facts': 'Canonical Facts',
+            'facts': 'Facts',
         }
-        labels = {k:v for k, v in labels.items() if k in columns}
-        ccsp_report_dataframe = ccsp_report_dataframe.rename(
-            columns=labels
-        )
+        labels = {k: v for k, v in labels.items() if k in columns}
+        ccsp_report_dataframe = ccsp_report_dataframe.rename(columns=labels)
 
         row_counter = 0
         rows = dataframe_to_rows(ccsp_report_dataframe, index=False)
@@ -291,37 +254,27 @@ def _build_data_section_scope(self, current_row, ws, dataframe, mode=None):
 
         return current_row + row_counter
 
-
     def _build_data_section_usage_by_node(self, current_row, ws, dataframe, mode=None, managed_node_type=None):
         for key, value in self.config['data_column_widths'].items():
             ws.column_dimensions[get_column_letter(key)].width = value
 
-        header_font = Font(name=self.FONT,
-                           size=10,
-                           color=self.BLACK_COLOR_HEX,
-                           bold=True)
-        value_font = Font(name=self.FONT,
-                          size=10,
-                          color=self.BLACK_COLOR_HEX)
+        header_font = Font(name=self.FONT, size=10, color=self.BLACK_COLOR_HEX, bold=True)
+        value_font = Font(name=self.FONT, size=10, color=self.BLACK_COLOR_HEX)
 
         agg_dict = {
-            "organizations": ('organization_name', 'nunique'),
-            "host_runs": ('host_name', 'count'),
-            "task_runs": ('task_runs', 'sum'),
-            "first_automation": ('first_automation', 'min'),
-            "last_automation": ('last_automation', 'max'),
-            "manage_node_types": ('manage_node_types', lambda x: merge_arrays(x)),
-            "events": ('events', lambda x: merge_arrays(x)),
-            "canonical_facts": ('canonical_facts', lambda x: merge_json_sets(x)),
-            "facts": ('facts', lambda x: merge_json_sets(x)),
+            'organizations': ('organization_name', 'nunique'),
+            'host_runs': ('host_name', 'count'),
+            'task_runs': ('task_runs', 'sum'),
+            'first_automation': ('first_automation', 'min'),
+            'last_automation': ('last_automation', 'max'),
+            'manage_node_types': ('manage_node_types', lambda x: merge_arrays(x)),
+            'events': ('events', lambda x: merge_arrays(x)),
+            'canonical_facts': ('canonical_facts', lambda x: merge_json_sets(x)),
+            'facts': ('facts', lambda x: merge_json_sets(x)),
         }
 
         # Now pass this dictionary into .agg()
-        ccsp_report_dataframe = (
-            dataframe
-            .groupby("host_name", dropna=False)
-            .agg(**agg_dict)
-        )
+        ccsp_report_dataframe = dataframe.groupby('host_name', dropna=False).agg(**agg_dict)
 
         # Convert arrays and dict fields into string, so they can be rendered into xlsx
         for col in ['manage_node_types', 'events', 'canonical_facts', 'facts']:
@@ -336,36 +289,34 @@ def _build_data_section_usage_by_node(self, current_row, ws, dataframe, mode=Non
             'first_automation',
             'last_automation',
         ]
-        if managed_node_type == "indirect":
+        if managed_node_type == 'indirect':
             columns += ['manage_node_types', 'canonical_facts', 'facts', 'events']
 
-        if mode == "by_organization":
+        if mode == 'by_organization':
             # Filter some columns out based on mode
             columns = [col for col in columns if col not in ['organizations']]
-        ccsp_report_dataframe = ccsp_report_dataframe.reindex(
-            columns=columns
-        )
+        ccsp_report_dataframe = ccsp_report_dataframe.reindex(columns=columns)
 
         labels = {
-            "host_name": "Host name",
-            "organizations": "Automated by\norganizations",
-            "host_runs": "Job runs",  # Job runs is the same as host_runs, Non-unique managed nodes automated
-            "task_runs": "Number of task\nruns",
-            'first_automation': "First\nautomation",
-            'last_automation': "Last\nautomation",
+            'host_name': self.HOST_NAME,
+            'organizations': 'Automated by\norganizations',
+            'host_runs': self.JOB_RUNS,  # Job runs is the same as host_runs, Non-unique managed nodes automated
+            'task_runs': self.NUM_OF_TASKS_OR_RUNS,
+            'first_automation': 'First\nautomation',
+            'last_automation': 'Last\nautomation',
         }
-        if managed_node_type == "indirect":
-            labels.update({
-                "manage_node_types": "Manage\nNode\nTypes",
-                "canonical_facts": "Canonical\nFacts",
-                "facts": "Facts",
-                "events": "Events",
-            })
-
-        labels = {k:v for k, v in labels.items() if k in columns}
-        ccsp_report_dataframe = ccsp_report_dataframe.rename(
-            columns=labels
-        )
+        if managed_node_type == 'indirect':
+            labels.update(
+                {
+                    'manage_node_types': 'Manage\nNode\nTypes',
+                    'canonical_facts': 'Canonical\nFacts',
+                    'facts': 'Facts',
+                    'events': 'Events',
+                }
+            )
+
+        labels = {k: v for k, v in labels.items() if k in columns}
+        ccsp_report_dataframe = ccsp_report_dataframe.rename(columns=labels)
 
         row_counter = 0
         rows = dataframe_to_rows(ccsp_report_dataframe, index=False)
@@ -391,33 +342,27 @@ def _build_data_section_usage_by_collections(self, current_row, ws, dataframe):
         for key, value in self.config['data_column_widths'].items():
             ws.column_dimensions[get_column_letter(key)].width = value
 
-        header_font = Font(name=self.FONT,
-                           size=10,
-                           color=self.BLACK_COLOR_HEX,
-                           bold=True)
-        value_font = Font(name=self.FONT,
-                          size=10,
-                          color=self.BLACK_COLOR_HEX)
+        header_font = Font(name=self.FONT, size=10, color=self.BLACK_COLOR_HEX, bold=True)
+        value_font = Font(name=self.FONT, size=10, color=self.BLACK_COLOR_HEX)
 
         # Take the content explorer dataframe and extract specific group by
-        ccsp_report_dataframe = dataframe.groupby(
-            ["collection_name"], dropna=False
-        ).agg(
+        ccsp_report_dataframe = dataframe.groupby(['collection_name'], dropna=False).agg(
             host_runs_unique=('host_name', 'nunique'),
             host_runs=('host_composite_id', 'nunique'),
             task_runs=('task_runs', 'sum'),
-            duration=('duration', "sum"))
+            duration=('duration', 'sum'),
+        )
 
         # Rename the columns based on the template
         ccsp_report_dataframe = ccsp_report_dataframe.reset_index()
 
         ccsp_report_dataframe = ccsp_report_dataframe.rename(
             columns={
-                "collection_name": "Collection name",
-                "host_runs_unique": "Unique managed nodes\nautomated",
-                "host_runs": "Non-unique managed\nnodes automated",
-                "task_runs": "Number of task\nruns",
-                "duration": "Duration of task\nruns [seconds]",
+                'collection_name': 'Collection name',
+                'host_runs_unique': self.HOST_RUNS_UNIQUE,
+                'host_runs': self.HOST_RUNS,
+                'task_runs': self.NUM_OF_TASKS_OR_RUNS,
+                'duration': self.DURATION,
             }
         )
 
@@ -445,33 +390,27 @@ def _build_data_section_usage_by_roles(self, current_row, ws, dataframe):
         for key, value in self.config['data_column_widths'].items():
             ws.column_dimensions[get_column_letter(key)].width = value
 
-        header_font = Font(name=self.FONT,
-                           size=10,
-                           color=self.BLACK_COLOR_HEX,
-                           bold=True)
-        value_font = Font(name=self.FONT,
-                          size=10,
-                          color=self.BLACK_COLOR_HEX)
+        header_font = Font(name=self.FONT, size=10, color=self.BLACK_COLOR_HEX, bold=True)
+        value_font = Font(name=self.FONT, size=10, color=self.BLACK_COLOR_HEX)
 
         # Take the content explorer dataframe and extract specific group by
-        ccsp_report_dataframe = dataframe.groupby(
-            ["role_name"], dropna=False
-        ).agg(
+        ccsp_report_dataframe = dataframe.groupby(['role_name'], dropna=False).agg(
             host_runs_unique=('host_name', 'nunique'),
             host_runs=('host_composite_id', 'nunique'),
             task_runs=('task_runs', 'sum'),
-            duration=('duration', "sum"))
+            duration=('duration', 'sum'),
+        )
 
         # Rename the columns based on the template
         ccsp_report_dataframe = ccsp_report_dataframe.reset_index()
 
         ccsp_report_dataframe = ccsp_report_dataframe.rename(
             columns={
-                "role_name": "Role name",
-                "host_runs_unique": "Unique managed nodes\nautomated",
-                "host_runs": "Non-unique managed\nnodes automated",
-                "task_runs": "Number of task\nruns",
-                "duration": "Duration of task\nruns [seconds]",
+                'role_name': 'Role name',
+                'host_runs_unique': self.HOST_RUNS_UNIQUE,
+                'host_runs': self.HOST_RUNS,
+                'task_runs': self.NUM_OF_TASKS_OR_RUNS,
+                'duration': self.DURATION,
             }
         )
 
@@ -500,32 +439,26 @@ def _build_data_section_usage_by_modules(self, current_row, ws, dataframe):
         for key, value in self.config['data_column_widths'].items():
             ws.column_dimensions[get_column_letter(key)].width = value
 
-        header_font = Font(name=self.FONT,
-                           size=10,
-                           color=self.BLACK_COLOR_HEX,
-                           bold=True)
-        value_font = Font(name=self.FONT,
-                          size=10,
-                          color=self.BLACK_COLOR_HEX)
-
-        ccsp_report_dataframe = dataframe.groupby(
-            ["module_name"], dropna=False
-        ).agg(
+        header_font = Font(name=self.FONT, size=10, color=self.BLACK_COLOR_HEX, bold=True)
+        value_font = Font(name=self.FONT, size=10, color=self.BLACK_COLOR_HEX)
+
+        ccsp_report_dataframe = dataframe.groupby(['module_name'], dropna=False).agg(
             host_runs_unique=('host_name', 'nunique'),
             host_runs=('host_composite_id', 'nunique'),
             task_runs=('task_runs', 'sum'),
-            duration=('duration', "sum"))
+            duration=('duration', 'sum'),
+        )
 
         # Rename the columns based on the template
         ccsp_report_dataframe = ccsp_report_dataframe.reset_index()
 
         ccsp_report_dataframe = ccsp_report_dataframe.rename(
             columns={
-                "module_name": "Module name",
-                "host_runs_unique": "Unique managed nodes\nautomated",
-                "host_runs": "Non-unique managed\nnodes automated",
-                "task_runs": "Number of task\nruns",
-                "duration": "Duration of task\nruns [seconds]",
+                'module_name': 'Module name',
+                'host_runs_unique': self.HOST_RUNS_UNIQUE,
+                'host_runs': self.HOST_RUNS,
+                'task_runs': self.NUM_OF_TASKS_OR_RUNS,
+                'duration': self.DURATION,
             }
         )
 
diff --git a/metrics_utility/management/commands/build_report.py b/metrics_utility/management/commands/build_report.py
index 8084d6382..d376da21d 100644
--- a/metrics_utility/management/commands/build_report.py
+++ b/metrics_utility/management/commands/build_report.py
@@ -7,23 +7,17 @@
 from django.core.management.base import BaseCommand
 from datetime import timezone
 
-from metrics_utility.automation_controller_billing.dataframe_engine.factory import \
-    Factory as DataframeEngineFactory
-from metrics_utility.automation_controller_billing.extract.factory import \
-    Factory as ExtractorFactory
-from metrics_utility.automation_controller_billing.helpers import \
-    parse_date_param
-from metrics_utility.automation_controller_billing.report.factory import \
-    Factory as ReportFactory
-from metrics_utility.automation_controller_billing.report_saver.factory import \
-    Factory as ReportSaverFactory
-from metrics_utility.exceptions import (BadRequiredEnvVar, BadShipTarget,
-                                        MissingRequiredEnvVar)
-from metrics_utility.management.validation import (
-    handle_directory_ship_target, handle_s3_ship_target)
+from metrics_utility.automation_controller_billing.dataframe_engine.factory import Factory as DataframeEngineFactory
+from metrics_utility.automation_controller_billing.extract.factory import Factory as ExtractorFactory
+from metrics_utility.automation_controller_billing.helpers import parse_date_param
+from metrics_utility.automation_controller_billing.report.factory import Factory as ReportFactory
+from metrics_utility.automation_controller_billing.report_saver.factory import Factory as ReportSaverFactory
+from metrics_utility.exceptions import BadRequiredEnvVar, BadShipTarget, MissingRequiredEnvVar
+from metrics_utility.management.validation import handle_directory_ship_target, handle_s3_ship_target
 
 from metrics_utility.metric_utils import get_optional_collectors
 
+
 class Command(BaseCommand):
     """
     Gather Automation Controller billing data
@@ -32,34 +26,34 @@ class Command(BaseCommand):
     help = 'Gather Automation Controller billing data'
 
     def add_arguments(self, parser):
-        parser.add_argument('--month',
-                            dest='month',
-                            action='store',
-                            help='Month the report will be generated for, with format YYYY-MM. '\
-                                 'If this params isn\'t provided, previou month report will be'\
-                                 'generated if it doesn\'t exists already.')
-        parser.add_argument('--since',
-                            dest='since',
-                            action='store',
-                            help='Date or number of days/months ago we want to generate the reports for.')
-        parser.add_argument('--until',
-                            dest='until',
-                            action='store',
-                            help='Date or number of days/months ago we want to generate the reports until. Not available for renewal guidance report')
-        parser.add_argument('--ephemeral',
-                            dest='ephemeral',
-                            action='store',
-                            help='Duration in months or days to determine if host is ephemeral. Months are taken'\
-                                 'as 30days duration.')
-        parser.add_argument('--force',
-                            dest='force',
-                            action='store_true',
-                            help='With this option, the existing reports will be overwritten if '\
-                                 'running this command again.')
-        parser.add_argument('--verbose',
-                            dest='verbose',
-                            action='store_true',
-                            help='Starts to print debug information to terminal.')
+        parser.add_argument(
+            '--month',
+            dest='month',
+            action='store',
+            help='Month the report will be generated for, with format YYYY-MM. '
+            "If this params isn't provided, previou month report will be"
+            "generated if it doesn't exists already.",
+        )
+        parser.add_argument('--since', dest='since', action='store', help='Date or number of days/months ago we want to generate the reports for.')
+        parser.add_argument(
+            '--until',
+            dest='until',
+            action='store',
+            help='Date or number of days/months ago we want to generate the reports until. Not available for renewal guidance report',
+        )
+        parser.add_argument(
+            '--ephemeral',
+            dest='ephemeral',
+            action='store',
+            help='Duration in months or days to determine if host is ephemeral. Months are takenas 30days duration.',
+        )
+        parser.add_argument(
+            '--force',
+            dest='force',
+            action='store_true',
+            help='With this option, the existing reports will be overwritten if running this command again.',
+        )
+        parser.add_argument('--verbose', dest='verbose', action='store_true', help='Starts to print debug information to terminal.')
 
     def init_logging(self):
         self.logger = logging.getLogger('awx.main.analytics')
@@ -104,29 +98,29 @@ def handle(self, *args, **options):
             extra_params['since_date'] = opt_since.date()
             extra_params['until_date'] = opt_until.date() if opt_until else now.date()
 
-            extra_params['report_period_range'] = f"{extra_params['since_date']}, {extra_params['until_date']}"
+            extra_params['report_period_range'] = f'{extra_params["since_date"]}, {extra_params["until_date"]}'
 
             extra_params['report_spreadsheet_destination_path'] = os.path.join(
                 extractor.get_report_path(extra_params['until_date']),
-                f"{extra_params['report_type']}-{opt_since.date()}--{extra_params['until_date']}.xlsx")
+                f'{extra_params["report_type"]}-{opt_since.date()}--{extra_params["until_date"]}.xlsx',
+            )
         else:
             extra_params['report_spreadsheet_destination_path'] = os.path.join(
-                extractor.get_report_path(month),
-                f"{extra_params['report_type']}-{opt_month}.xlsx")
+                extractor.get_report_path(month), f'{extra_params["report_type"]}-{opt_month}.xlsx'
+            )
 
         report_saver_engine = ReportSaverFactory(ship_target, extra_params=extra_params).create()
 
         if report_saver_engine.report_exist() and not opt_force:
             # If the monthly report already exists, skip the generation
-            self.logger.info("Skipping report generation, report: "\
-                             f"{report_saver_engine.report_spreadsheet_destination_path} already exists. "\
-                             "Use --force option to override the report.")
+            self.logger.info(
+                'Skipping report generation, report: '
+                f'{report_saver_engine.report_spreadsheet_destination_path} already exists. '
+                'Use --force option to override the report.'
+            )
             return
 
-        report_dataframe = DataframeEngineFactory(
-            extractor=extractor,
-            month=month,
-            extra_params=extra_params).create()
+        report_dataframe = DataframeEngineFactory(extractor=extractor, month=month, extra_params=extra_params).create()
 
         if report_dataframe[0] is None or report_dataframe[0].empty:
             if opt_since is not None:
@@ -135,27 +129,28 @@ def handle(self, *args, **options):
                 self.logger.info(f'No billing data for month {opt_month}')
             return
 
-        report_engine = ReportFactory(report_period=opt_month,
-                                      report_dataframe=report_dataframe,
-                                      ship_target=ship_target,
-                                      extra_params=extra_params).create()
+        report_engine = ReportFactory(
+            report_period=opt_month, report_dataframe=report_dataframe, ship_target=ship_target, extra_params=extra_params
+        ).create()
         report_spreadsheet = report_engine.build_spreadsheet()
 
         # Save the report to the configured destination
         report_saver_engine.save(report_spreadsheet)
-        self.logger.info(f"Report generated into {ship_target}: {report_saver_engine.report_spreadsheet_destination_path}")
+        self.logger.info(f'Report generated into {ship_target}: {report_saver_engine.report_spreadsheet_destination_path}')
 
     def _handle_ship_target(self, ship_target):
-        if ship_target == "controller_db":
+        if ship_target == 'controller_db':
             return {}
-        elif ship_target == "directory":
+        elif ship_target == 'directory':
             return handle_directory_ship_target(ship_target)
-        elif ship_target == "s3":
+        elif ship_target == 's3':
             return handle_s3_ship_target(ship_target)
         else:
-            raise BadShipTarget("Unexpected value for METRICS_UTILITY_SHIP_TARGET env var"\
-                                ", allowed value for local report generation are "\
-                                "['s3', 'directory', 'controller_db']")
+            raise BadShipTarget(
+                'Unexpected value for METRICS_UTILITY_SHIP_TARGET env var'
+                ', allowed value for local report generation are '
+                "['s3', 'directory', 'controller_db']"
+            )
 
     def _handle_extra_params(self, ship_target=None):
         base = self._handle_ship_target(ship_target)
@@ -166,59 +161,64 @@ def _handle_extra_params(self, ship_target=None):
 
         if not ship_path:
             raise MissingRequiredEnvVar(
-                "Missing required env variable METRICS_UTILITY_SHIP_PATH, having destination"\
-                " for the generated data and for the built reports")
+                'Missing required env variable METRICS_UTILITY_SHIP_PATH, having destination for the generated data and for the built reports'
+            )
 
         if not report_type:
-            raise MissingRequiredEnvVar(
-                "Missing required env variable METRICS_UTILITY_REPORT_TYPE.")
+            raise MissingRequiredEnvVar('Missing required env variable METRICS_UTILITY_REPORT_TYPE.')
         elif report_type not in ['CCSP', 'CCSPv2', 'RENEWAL_GUIDANCE']:
             raise BadRequiredEnvVar(
-                "Bad value for required env variable METRICS_UTILITY_REPORT_TYPE, allowed"\
-                " values are: ['CCSP', 'CCSPv2', 'RENEWAL_GUIDANCE']")
-
-        base.update({
-            "ship_path": ship_path,
-            "report_type": report_type,
-            "price_per_node": price_per_node,
-            # XLSX specific params
-            "report_sku": os.getenv('METRICS_UTILITY_REPORT_SKU', ""),
-            "report_sku_description": os.getenv('METRICS_UTILITY_REPORT_SKU_DESCRIPTION', ""),
-            "report_h1_heading": os.getenv('METRICS_UTILITY_REPORT_H1_HEADING', ""),
-            "report_company_name": os.getenv('METRICS_UTILITY_REPORT_COMPANY_NAME', ""),
-            "report_email": os.getenv('METRICS_UTILITY_REPORT_EMAIL', ""),
-            "report_rhn_login": os.getenv('METRICS_UTILITY_REPORT_RHN_LOGIN', ""),
-            "report_po_number": os.getenv('METRICS_UTILITY_REPORT_PO_NUMBER', ""),
-            "report_company_business_leader": os.getenv('METRICS_UTILITY_REPORT_COMPANY_BUSINESS_LEADER', ""),
-            "report_company_procurement_leader": os.getenv('METRICS_UTILITY_REPORT_COMPANY_PROCUREMENT_LEADER', ""),
-            "report_end_user_company_name": os.getenv('METRICS_UTILITY_REPORT_END_USER_COMPANY_NAME', ""),
-            "report_end_user_company_city": os.getenv('METRICS_UTILITY_REPORT_END_USER_CITY', ""),
-            "report_end_user_company_state": os.getenv('METRICS_UTILITY_REPORT_END_USER_STATE', ""),
-            "report_end_user_company_country": os.getenv('METRICS_UTILITY_REPORT_END_USER_COUNTRY', ""),
-            # Renewal guidance specific params
-            "report_renewal_guidance_dedup_iterations": os.getenv('REPORT_RENEWAL_GUIDANCE_DEDUP_ITERATIONS', "3"),
-            "report_organization_filter": os.getenv('METRICS_UTILITY_ORGANIZATION_FILTER', None),
-            # S3 specific options
-            "s3_bucket_name": os.getenv('METRICS_UTILITY_BUCKET_NAME', None),
-            "s3_bucket_endpoint": os.getenv('METRICS_UTILITY_BUCKET_ENDPOINT', None),
-            "s3_bucket_region": os.getenv('METRICS_UTILITY_BUCKET_REGION', None),
-            "s3_bucket_id": os.getenv('METRICS_UTILITY_BUCKET_ID', None),
-            "s3_bucket_key": os.getenv('METRICS_UTILITY_BUCKET_KEY', None),
-            "optional_collectors" : get_optional_collectors(),
-        })
+                "Bad value for required env variable METRICS_UTILITY_REPORT_TYPE, allowed values are: ['CCSP', 'CCSPv2', 'RENEWAL_GUIDANCE']"
+            )
+
+        base.update(
+            {
+                'ship_path': ship_path,
+                'report_type': report_type,
+                'price_per_node': price_per_node,
+                # XLSX specific params
+                'report_sku': os.getenv('METRICS_UTILITY_REPORT_SKU', ''),
+                'report_sku_description': os.getenv('METRICS_UTILITY_REPORT_SKU_DESCRIPTION', ''),
+                'report_h1_heading': os.getenv('METRICS_UTILITY_REPORT_H1_HEADING', ''),
+                'report_company_name': os.getenv('METRICS_UTILITY_REPORT_COMPANY_NAME', ''),
+                'report_email': os.getenv('METRICS_UTILITY_REPORT_EMAIL', ''),
+                'report_rhn_login': os.getenv('METRICS_UTILITY_REPORT_RHN_LOGIN', ''),
+                'report_po_number': os.getenv('METRICS_UTILITY_REPORT_PO_NUMBER', ''),
+                'report_company_business_leader': os.getenv('METRICS_UTILITY_REPORT_COMPANY_BUSINESS_LEADER', ''),
+                'report_company_procurement_leader': os.getenv('METRICS_UTILITY_REPORT_COMPANY_PROCUREMENT_LEADER', ''),
+                'report_end_user_company_name': os.getenv('METRICS_UTILITY_REPORT_END_USER_COMPANY_NAME', ''),
+                'report_end_user_company_city': os.getenv('METRICS_UTILITY_REPORT_END_USER_CITY', ''),
+                'report_end_user_company_state': os.getenv('METRICS_UTILITY_REPORT_END_USER_STATE', ''),
+                'report_end_user_company_country': os.getenv('METRICS_UTILITY_REPORT_END_USER_COUNTRY', ''),
+                # Renewal guidance specific params
+                'report_renewal_guidance_dedup_iterations': os.getenv('REPORT_RENEWAL_GUIDANCE_DEDUP_ITERATIONS', '3'),
+                'report_organization_filter': os.getenv('METRICS_UTILITY_ORGANIZATION_FILTER', None),
+                # S3 specific options
+                's3_bucket_name': os.getenv('METRICS_UTILITY_BUCKET_NAME', None),
+                's3_bucket_endpoint': os.getenv('METRICS_UTILITY_BUCKET_ENDPOINT', None),
+                's3_bucket_region': os.getenv('METRICS_UTILITY_BUCKET_REGION', None),
+                's3_bucket_id': os.getenv('METRICS_UTILITY_BUCKET_ID', None),
+                's3_bucket_key': os.getenv('METRICS_UTILITY_BUCKET_KEY', None),
+                'optional_collectors': get_optional_collectors(),
+                'optional_sheets': os.getenv(
+                    'METRICS_UTILITY_OPTIONAL_CCSP_REPORT_SHEETS',
+                    'ccsp_summary,managed_nodes,usage_by_organizations,usage_by_collections,usage_by_roles,usage_by_modules',
+                ).split(','),
+            }
+        )
         return base
 
     def _handle_month(self, month):
         # Process month argument
         if month is not None:
-            date = date_parse(f"{month}-01")
+            date = date_parse(f'{month}-01')
         else:
             # Return last month if no month was passed
             beginning_of_the_month = datetime.date.today().replace(day=1)
             beginning_of_the_previous_month = beginning_of_the_month - relativedelta(months=1)
             date = beginning_of_the_previous_month
-            y = date.strftime("%Y")
-            m = date.strftime("%m")
-            month = f"{y}-{m}"
+            y = date.strftime('%Y')
+            m = date.strftime('%m')
+            month = f'{y}-{m}'
 
         return month, date


Reviews:
user: login: himdel
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-03-11T16:20:58Z
user: login: himdel
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-03-13T12:49:17Z
user: login: Ladas
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-03-17T14:19:12Z
user: login: himdel
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-03-17T21:51:20Z
user: login: himdel
state: APPROVED
author_association: CONTRIBUTOR
submitted_at: 2025-03-18T14:15:35Z
user: login: Ladas
body: Looks great!
state: APPROVED
author_association: CONTRIBUTOR
submitted_at: 2025-03-18T14:35:20Z

Review comments:
diff_hunk: @@ -87,29 +87,40 @@ def process_tarballs(self, path, temp_dir):
     # for index, row in chunk.iterrows():
     #     print(row)
 
-    if os.path.exists(os.path.join(temp_dir, 'job_host_summary.csv')):
-        job_host_summary = pd.read_csv(os.path.join(temp_dir, 'job_host_summary.csv'))
-    else:
-        job_host_summary = pd.DataFrame([{}])
-
-    if os.path.exists(os.path.join(temp_dir, 'main_indirectmanagednodeaudit.csv')):
-        indirect_nodes = pd.read_csv(os.path.join(temp_dir, 'main_indirectmanagednodeaudit.csv'))
-    else:
-        indirect_nodes = pd.DataFrame([{}])
-
-    if os.path.exists(os.path.join(temp_dir, 'main_jobevent.csv')):
-        main_jobevent = pd.read_csv(os.path.join(temp_dir, 'main_jobevent.csv'))
-    else:
-        main_jobevent = pd.DataFrame([{}])
-
-    if os.path.exists(os.path.join(temp_dir, 'main_host.csv')):
-        main_host = pd.read_csv(os.path.join(temp_dir, 'main_host.csv'))
-    else:
-        main_host = pd.DataFrame([{}])
-
-
-    return {'main_jobevent': main_jobevent,
-            'main_host': main_host,
-            'job_host_summary': job_host_summary,
-            'indirect_nodes' : indirect_nodes,
-            'config': config}
+    needed_data = {'config': config}
+    if required_sheets(self,['ccsp_summary','managed_nodes_by_organizations','managed_nodes','usage_by_organizations']):
+        needed_data['job_host_summary'] = build_data_batch(temp_dir,'job_host_summary')
+    if required_sheets(self,['usage_by_collections','usage_by_roles','usage_by_modules']):
+        needed_data['main_jobevent'] = build_data_batch(temp_dir,'main_jobevent')
+    if required_sheets(self,['indirectly_managed_nodes']):
+        needed_data['indirect_nodes']= build_data_batch(temp_dir,'indirect_nodes')
+    if required_sheets(self,['inventory_scope']):
+        needed_data['main_host']= build_data_batch(temp_dir,'main_host')
+
+    return needed_data
+
+
+def build_data_batch(temp_dir, file_name):
+      """
+      Builds the report with only the necessary sheets.
+      """
+
+      if os.path.exists(os.path.join(temp_dir, f'{file_name}.csv')):
+          return pd.read_csv(os.path.join(temp_dir, f'{file_name}.csv'))
+      else:
+          return pd.DataFrame([{}])
+
+
+def required_sheets(self, sheets_required):
+    """
+    Checks for METRICS_UTILITY_OPTIONAL_CCSP_REPORT_SHEETS values and return those values.
+    If none found, give it the default CCSP report sheet options.
+    Returns a boolean so we know which sheets to provide in the report.
+
+    """
+    sheet_options = os.getenv('METRICS_UTILITY_OPTIONAL_CCSP_REPORT_SHEETS')
+    if sheet_options is None:
+        sheet_options = self.optional_report_sheets()
path: metrics_utility/automation_controller_billing/extract/extractor_common.py
user: login: himdel
body: one of these returns a comma-separated string, the other returns an array of strings

```suggestion
    sheet_options = self.optional_report_sheets()
```
created_at: 2025-03-11T16:20:58Z
author_association: CONTRIBUTOR
diff_hunk: @@ -87,29 +87,53 @@ def process_tarballs(self, path, temp_dir):
     # for index, row in chunk.iterrows():
     #     print(row)
 
-    if os.path.exists(os.path.join(temp_dir, 'job_host_summary.csv')):
-        job_host_summary = pd.read_csv(os.path.join(temp_dir, 'job_host_summary.csv'))
-    else:
-        job_host_summary = pd.DataFrame([{}])
-
-    if os.path.exists(os.path.join(temp_dir, 'main_indirectmanagednodeaudit.csv')):
-        indirect_nodes = pd.read_csv(os.path.join(temp_dir, 'main_indirectmanagednodeaudit.csv'))
-    else:
-        indirect_nodes = pd.DataFrame([{}])
-
-    if os.path.exists(os.path.join(temp_dir, 'main_jobevent.csv')):
-        main_jobevent = pd.read_csv(os.path.join(temp_dir, 'main_jobevent.csv'))
-    else:
-        main_jobevent = pd.DataFrame([{}])
-
-    if os.path.exists(os.path.join(temp_dir, 'main_host.csv')):
-        main_host = pd.read_csv(os.path.join(temp_dir, 'main_host.csv'))
-    else:
-        main_host = pd.DataFrame([{}])
-
-
-    return {'main_jobevent': main_jobevent,
-            'main_host': main_host,
-            'job_host_summary': job_host_summary,
-            'indirect_nodes' : indirect_nodes,
-            'config': config}
+    empty_dataframe = pd.DataFrame([{}])
+    needed_data = {
+        'config': config,
+        'job_host_summary':empty_dataframe,
+        'indirect_nodes':empty_dataframe,
+        'main_jobevent':empty_dataframe,
+        'main_host':empty_dataframe
+    }
+    if required_sheets(self,['ccsp_summary','managed_nodes','indirectly_managed_nodes','inventory_scope',
+                             'usage_by_organizations','managed_nodes_by_organizations']):
+        needed_data['job_host_summary'] = build_data_batch(temp_dir,'job_host_summary')
+
+    if required_sheets(self,['managed_nodes', 'indirectly_managed_nodes', 'usage_by_organizations']):
+        needed_data['indirect_nodes']= build_data_batch(temp_dir,'main_indirectmanagednodeaudit')
+
+    if required_sheets(self,['usage_by_collections','usage_by_roles','usage_by_modules','usage_by_organizations']):
+        needed_data['main_jobevent'] = build_data_batch(temp_dir,'main_jobevent')
+
+    if required_sheets(self,['jobs','managed_nodes_by_organizations','managed_nodes','inventory_scope',
+                             'usage_by_collections,usage_by_roles,usage_by_modules','managed_nodes']):
+        needed_data['main_host']= build_data_batch(temp_dir,'main_host')
+
+    return needed_data
+
+
+def build_data_batch(temp_dir, file_name):
+      """
+      Builds the report with only the necessary sheets.
+      """
+
+      if os.path.exists(os.path.join(temp_dir, f'{file_name}.csv')):
+        return pd.read_csv(os.path.join(temp_dir, f'{file_name}.csv'))
+      else:
+        return pd.DataFrame([{}])
+
+
+def required_sheets(self, sheets_required):
+    """
+    Checks for METRICS_UTILITY_OPTIONAL_CCSP_REPORT_SHEETS values and return those values.
+    If none found, give it the default CCSP report sheet options.
+    Returns a boolean so we know which sheets to provide in the report.
+
+    """
+    sheet_options = os.getenv('METRICS_UTILITY_OPTIONAL_CCSP_REPORT_SHEETS','ccsp_summary,managed_nodes,\
+                              indirectly_managed_nodes,inventory_scope,usage_by_organizations,\
+                              managed_nodes_by_organizations').split(',')
+    if sheet_options is None: ## We might not need this..split(',')
+        sheet_options = self.optional_report_sheets()
path: metrics_utility/automation_controller_billing/extract/extractor_common.py
user: login: himdel
body: ```suggestion
    sheet_options = self.optional_report_sheets()
```

the first part does exactly what the function does, so let's just keep the function call?
created_at: 2025-03-13T12:49:17Z
author_association: CONTRIBUTOR
diff_hunk: @@ -3,7 +3,78 @@
 import tarfile
 
 
-def write_member(member_path, file_obj, max_size, total_extracted_size):
+class Base:
path: metrics_utility/automation_controller_billing/extract/extractor_common.py
user: login: Ladas
body: we should ideally call this based on the file name, we can rename the file to base, to follow the pattern of the other factories?
created_at: 2025-03-17T14:19:12Z
author_association: CONTRIBUTOR
diff_hunk: @@ -0,0 +1,147 @@
+import pandas as pd
+import os
+import tarfile
+
+
+class Base:
+    def process_tarballs(self, path, temp_dir):
+        _safe_extract(path, temp_dir)
+        config = self.load_config(os.path.join(temp_dir, 'config.json'))
+
+        # # TODO: read the csvs in batches
+        # for chunk in pd.read_csv(filename, chunksize=chunksize):
+        # # chunk is a DataFrame. To "process" the rows in the chunk:
+        # for index, row in chunk.iterrows():
+        #     print(row)
+
+        empty_dataframe = pd.DataFrame([{}])
+        needed_data = {
+            'config': config,
+            'job_host_summary': empty_dataframe,
+            'indirect_nodes': empty_dataframe,
+            'main_jobevent': empty_dataframe,
+            'main_host': empty_dataframe,
+        }
+        if self.required_sheets(
+            [
+                'ccsp_summary',
+                'managed_nodes',
+                'indirectly_managed_nodes',
+                'inventory_scope',
+                'usage_by_organizations',
+                'managed_nodes_by_organizations',
+            ]
+        ):
+            needed_data['job_host_summary'] = self.build_data_batch(temp_dir, 'job_host_summary')
+
+        if self.required_sheets(['managed_nodes', 'indirectly_managed_nodes', 'usage_by_organizations']):
+            needed_data['indirect_nodes'] = self.build_data_batch(temp_dir, 'main_indirectmanagednodeaudit')
+
+        if self.required_sheets(['usage_by_collections', 'usage_by_roles', 'usage_by_modules', 'usage_by_organizations']):
+            needed_data['main_jobevent'] = self.build_data_batch(temp_dir, 'main_jobevent')
+
+        if self.required_sheets(
+            [
+                'jobs',
+                'managed_nodes_by_organizations',
+                'managed_nodes',
+                'inventory_scope',
+                'usage_by_collections,usage_by_roles,usage_by_modules',
path: metrics_utility/automation_controller_billing/extract/base.py
user: login: himdel
body: ```suggestion
                'usage_by_collections',
                'usage_by_roles',
                'usage_by_modules',
```
created_at: 2025-03-17T21:51:20Z
author_association: CONTRIBUTOR

PR comments:
user: login: himdel
created_at: 2025-03-14T17:30:09Z
body: Can you run `uvx ruff format ` on the changed files please? It should fix the indentation levels :) 


author_association: CONTRIBUTOR
user: login: sonarqubecloud[bot]
created_at: 2025-03-17T14:37:48Z
body: ## [![Quality Gate Passed](https://sonarsource.github.io/sonarcloud-github-static-resources/v2/checks/QualityGateBadge/qg-passed-20px.png 'Quality Gate Passed')](https://sonarcloud.io/dashboard?id=ansible_metrics-utility&pullRequest=86) **Quality Gate passed**  
Issues  
![](https://sonarsource.github.io/sonarcloud-github-static-resources/v2/common/passed.svg '') [0 New issues](https://sonarcloud.io/project/issues?id=ansible_metrics-utility&pullRequest=86&issueStatuses=OPEN,CONFIRMED&sinceLeakPeriod=true)  
![](https://sonarsource.github.io/sonarcloud-github-static-resources/v2/common/accepted.svg '') [0 Accepted issues](https://sonarcloud.io/project/issues?id=ansible_metrics-utility&pullRequest=86&issueStatuses=ACCEPTED)

Measures  
![](https://sonarsource.github.io/sonarcloud-github-static-resources/v2/common/passed.svg '') [0 Security Hotspots](https://sonarcloud.io/project/security_hotspots?id=ansible_metrics-utility&pullRequest=86&issueStatuses=OPEN,CONFIRMED&sinceLeakPeriod=true)  
![](https://sonarsource.github.io/sonarcloud-github-static-resources/v2/common/no-data.svg '') No data about Coverage  
![](https://sonarsource.github.io/sonarcloud-github-static-resources/v2/common/passed.svg '') [0.0% Duplication on New Code](https://sonarcloud.io/component_measures?id=ansible_metrics-utility&pullRequest=86&metric=new_duplicated_lines_density&view=list)  
  
[See analysis details on SonarQube Cloud](https://sonarcloud.io/dashboard?id=ansible_metrics-utility&pullRequest=86)


author_association: NONE
user: login: himdel
created_at: 2025-03-17T22:02:50Z
body: Reading the code, this is my understanding of which sheets map to which CSVs, if it helps anyone :)

````python
SHEET2CSV = {
    'ccsp_summary': ['job_host_summary'],
    'indirectly_managed_nodes': ['job_host_summary', 'main_indirectmanagednodeaudit'],
    'inventory_scope': ['job_host_summary', 'main_host'],
    'jobs': ['main_host'],
    'managed_nodes': ['job_host_summary', 'main_host', 'main_indirectmanagednodeaudit'],
    'managed_nodes_by_organizations': ['job_host_summary', 'main_host'],
    'usage_by_collections': ['main_host', 'main_jobevent'],
    'usage_by_modules': ['main_host', 'main_jobevent'],
    'usage_by_organizations': ['job_host_summary', 'main_indirectmanagednodeaudit', 'main_jobevent'],
    'usage_by_roles': ['main_host', 'main_jobevent'],
}
```
author_association: CONTRIBUTOR