Repository: softwareone-platform/ffc-finops-operations
Pull request #179: feat: make generation of monthly charges more memory effecient (MPT-9570)
Author: login: arturbalabanov
Created: 2025-05-02T09:35:55Z | Merged: 2025-05-02T16:55:18Z

After many headaches with pandas I gave up on using it since using it in append mode is just a wrapper around openpyxl (other xlsx engines don't support append mode) but with more annoying interface.

With the `write_only` mode openpyxls guarantees max 10MB memory usage for _any_ file. In addition, we're now streaming the datasource expenses instead of holding the full list in memory, so the whole script is very scalable memory-wise.

With that being said, we're still generating the files one account at a time and re-fetching the datasource expenses for each of them (even though for the same currency the results will be the same). I decided to go with this approach because:

1. We don't have big restrictions for the runtime of the script, so even if it's a bit slow that should be fine and want to avoid premature optimisations
2. The code is simpler and easier to follow this way. Otherwise we need to keep track of multiple files (some of them potentially empty, extract the upload at the very end (so that it's not done in a db transaction) and similar workarounds for the charge file db records.
3. With the current approach it will be significantly easier to implement the other additions we're planning for the script


Diff:
diff --git a/app/commands/generate_monthly_charges.py b/app/commands/generate_monthly_charges.py
index 5801f868..62bd0ee5 100644
--- a/app/commands/generate_monthly_charges.py
+++ b/app/commands/generate_monthly_charges.py
@@ -2,14 +2,14 @@
 import logging
 import pathlib
 import zipfile
-from collections.abc import Sequence
+from collections.abc import AsyncGenerator, Sequence
 from copy import copy
 from dataclasses import dataclass
 from datetime import UTC, date, datetime
 from decimal import Decimal
 from typing import Annotated, Self
 
-import pandas as pd
+import openpyxl
 import typer
 from azure.core.exceptions import AzureError, ClientAuthenticationError, ResourceNotFoundError
 from dateutil.relativedelta import relativedelta  # type: ignore[import-untyped]
@@ -88,123 +88,106 @@ def contra_entry(self) -> Self:
         return contra_entry
 
 
-@dataclass
 class ChargesFileGenerator:
-    account: Account
-    currency: str
-    currency_converter: CurrencyConverter
-    exports_dir: pathlib.Path
-
-    def get_charge_entries(
-        self, datasource_expenses: Sequence[DatasourceExpense]
-    ) -> list[ChargeEntry]:
-        return [
-            charge_entry
-            for ds_exp in datasource_expenses
-            for charge_entry in self.charge_entries_for_datasource_expense(ds_exp)
-        ]
-
-    def charge_entries_for_datasource_expense(
-        self, datasource_expense: DatasourceExpense
-    ) -> list[ChargeEntry]:
+    def __init__(
+        self,
+        account: Account,
+        currency: str,
+        currency_converter: CurrencyConverter,
+        exports_dir: pathlib.Path,
+    ) -> None:
+        self.account = account
+        self.currency = currency
+        self.currency_converter = currency_converter
+        self.exports_dir = exports_dir
+        self.workbook = openpyxl.Workbook(write_only=True)
+        self.worksheet = self.workbook.create_sheet()
+        self.running_total = Decimal(0)
+        self.total_rows = 0
+        self.source_currencies: set[str] = set()
+
+    @property
+    def has_entries(self) -> bool:
+        return self.total_rows > 0
+
+    def append_row(self, charge_entry: ChargeEntry) -> None:
+        row = {
+            "Entry ID": self.total_rows if self.has_entries else 1,
+            "Subscription Search Criteria": "subscription.externalIds.vendor",
+            "Subscription Search Value": charge_entry.subscription_search_value,
+            "Item Search Criteria": "item.externalIds.vendor",
+            "Item Search Value": charge_entry.item_search_value,
+            "Usage Start Time": charge_entry.usage_start_time.strftime("%-d-%b-%Y"),
+            "Usage End Time": charge_entry.usage_end_time.strftime("%-d-%b-%Y"),
+            "Quantity": 1,
+            "Purchase Price": charge_entry.price.quantize(Decimal("0.01")),
+            "Total Purchase Price": charge_entry.price.quantize(Decimal("0.01")),
+            "External Reference": charge_entry.external_reference,
+            "Vendor Description 1": charge_entry.vendor_description_1,
+            "Vendor Description 2": charge_entry.vendor_description_2,
+            "Vendor Reference": "",
+        }
+
+        if not self.has_entries:
+            # Add the header row
+            self.worksheet.append(list(row.keys()))
+            self.total_rows += 1
+
+        self.worksheet.append(list(row.values()))
+        self.total_rows += 1
+        self.running_total += charge_entry.price
+
+    def add_datasource_expense(self, datasource_expense: DatasourceExpense) -> None:
+        self.source_currencies.add(datasource_expense.organization.currency)
         entry = ChargeEntry.from_datasource_expense(datasource_expense, self.currency_converter)
 
         if self.account.type == AccountType.AFFILIATE:
             for entitlement in datasource_expense.entitlements:
-                if entitlement.owner != self.account:
+                if (
+                    entitlement.owner != self.account
+                    or entitlement.status != EntitlementStatus.ACTIVE
+                ):
                     continue
 
-                if entitlement.status != EntitlementStatus.ACTIVE:
-                    continue
+                self.append_row(entry)
 
                 # If multiple entitlements match the same datasource expense, we need to
                 # return only the first one (the entitlements are already ordered by
                 # created_at)
-                return [entry]
+                return
 
-            return []
+            return
 
         if self.account.type == AccountType.OPERATIONS:
-            charge_entries = [entry]
+            self.append_row(entry)
 
             for entitlement in datasource_expense.entitlements:
                 if entitlement.status != EntitlementStatus.ACTIVE:
                     continue
 
-                charge_entries.append(entry.contra_entry())
+                self.append_row(entry.contra_entry())
 
                 # If multiple entitlements match the same datasource expense, we need to
                 # add only the contra entry for the first one (the entitlements are
                 # already ordered by created_at)
-                break
+                return
 
-            return charge_entries
+            return
 
         raise ValueError(f"Unknown account type: {self.account.type}")  # pragma: no cover
 
-    def generate_charges_file_dataframe(
-        self, datasource_expenses: Sequence[DatasourceExpense]
-    ) -> pd.DataFrame | None:
+    def save(self, filename: str) -> pathlib.Path:
         logger.info(
-            "Generating charges file contents for account %s and currency %s",
-            self.account.id,
-            self.currency,
-        )
-
-        charge_entries = self.get_charge_entries(datasource_expenses)
-
-        if not charge_entries:
-            logger.warning(
-                "No charge entries found for account %s and currency %s, "
-                "skipping generating charges file as it would be empty",
-                self.account.id,
-                self.currency,
-            )
-            return None
-
-        df = pd.DataFrame(
-            [
-                {
-                    "Entry ID": row_num,
-                    "Subscription Search Criteria": "subscription.externalIds.vendor",
-                    "Subscription Search Value": entry.subscription_search_value,
-                    "Item Search Criteria": "item.externalIds.vendor",
-                    "Item Search Value": entry.item_search_value,
-                    "Usage Start Time": entry.usage_start_time.strftime("%-d-%b-%Y"),
-                    "Usage End Time": entry.usage_end_time.strftime("%-d-%b-%Y"),
-                    "Quantity": 1,
-                    "Purchase Price": entry.price.quantize(Decimal("0.01")),
-                    "Total Purchase Price": entry.price.quantize(Decimal("0.01")),
-                    "External Reference": entry.external_reference,
-                    "Vendor Description 1": entry.vendor_description_1,
-                    "Vendor Description 2": entry.vendor_description_2,
-                    "Vendor Reference": "",
-                }
-                for row_num, entry in enumerate(charge_entries, start=1)
-            ]
-        )
-
-        logger.info(
-            "Charges file contents for account %s and currency %s generated successfully",
-            self.account.id,
-            self.currency,
-        )
-
-        return df
-
-    def export_to_excel(self, df: pd.DataFrame, filename: str) -> pathlib.Path:
-        logger.info(
-            "Exporting charge file contents for account %s and currency %s to excel format",
+            "Exporting charge file contents for account %s and currency %s to Excel format",
             self.account.id,
             self.currency,
         )
         filepath = self.exports_dir / filename
 
-        df.to_excel(filepath, header=True, index=False)
+        self.workbook.save(filepath)
 
         logger.info(
-            "Charge file contents for account %s and currency %s "
-            "exported to excel format and saved at %s",
+            "Charges file generated for account %s and currency %s and saved to %s",
             self.account.id,
             self.currency,
             str(filepath.resolve()),
@@ -212,8 +195,9 @@ def export_to_excel(self, df: pd.DataFrame, filename: str) -> pathlib.Path:
 
         return filepath
 
-    def export_to_zip(self, df: pd.DataFrame, filename: str) -> pathlib.Path:
-        filepath = self.exports_dir / filename
+    def make_archive(self, filename: str) -> pathlib.Path:
+        if not self.has_entries:
+            raise ValueError("Cannot export to zip format: no records found in the charges file")
 
         logger.info(
             "Exporting charge file contents together with the currency conversion rates "
@@ -222,14 +206,20 @@ def export_to_zip(self, df: pd.DataFrame, filename: str) -> pathlib.Path:
             self.currency,
         )
 
-        excel_filepath = self.export_to_excel(df, filename="charges.xlsx")
+        excel_filepath = self.save("charges.xlsx")
+        filepath = self.exports_dir / filename
 
         with zipfile.ZipFile(filepath, mode="w") as archive:
-            archive.write(excel_filepath, arcname=excel_filepath.name)
-            archive.writestr(
-                f"exchange_rates_{self.currency}.json",
-                self.currency_converter.get_exchangerate_api_response_json(self.currency),
-            )
+            archive.write(excel_filepath, arcname="charges.xlsx")
+
+            for source_currency in self.source_currencies:
+                if source_currency == self.currency:
+                    continue
+
+                archive.writestr(
+                    f"exchange_rates_{source_currency}.json",
+                    self.currency_converter.get_exchangerate_api_response_json(source_currency),
+                )
 
         excel_filepath.unlink(missing_ok=True)
 
@@ -242,10 +232,6 @@ def export_to_zip(self, df: pd.DataFrame, filename: str) -> pathlib.Path:
 
         return filepath
 
-    @staticmethod
-    def get_total_amount(df: pd.DataFrame) -> Decimal:
-        return df["Total Purchase Price"].sum()
-
 
 async def upload_charges_file_to_azure(charges_file: ChargesFile, filepath: pathlib.Path) -> bool:
     logging.info(
@@ -333,25 +319,21 @@ async def fetch_existing_generated_charges_file(
 
 
 async def fetch_datasource_expenses(
-    session: AsyncSession, account: Account, currency: str
-) -> Sequence[DatasourceExpense]:
+    session: AsyncSession, currency: str
+) -> AsyncGenerator[DatasourceExpense]:
     today = datetime.now(UTC).date()
     last_month = today - relativedelta(months=1)
 
     organization_handler = OrganizationHandler(session)
     datasource_expense_handler = DatasourceExpenseHandler(session)
 
-    logger.info(
-        "Querying organizations to process for account %s with billing currency = %s",
-        account.id,
-        currency,
-    )
+    logger.info("Querying organizations to process with billing currency = %s", currency)
 
     organizations = await organization_handler.query_db(
         where_clauses=[Organization.billing_currency == currency]
     )
     logger.info(
-        "Found %d organizations to process for billing currency %s",
+        "Found %d organizations to process with billing currency %s",
         len(organizations),
         currency,
     )
@@ -363,24 +345,16 @@ async def fetch_datasource_expenses(
         last_month.month,
         last_month.year,
     )
-    all_orgs_expenses = await datasource_expense_handler.query_db(
-        where_clauses=[
+    all_orgs_expenses = datasource_expense_handler.stream_scalars(
+        extra_conditions=[
             DatasourceExpense.organization_id.in_(org.id for org in organizations),
             DatasourceExpense.month == last_month.month,
             DatasourceExpense.year == last_month.year,
         ],
-        unique=True,
-    )
-    logger.info(
-        "Found %d datasource expenses for all organizations "
-        "with %s billing currency for month = %s, year = %s",
-        len(all_orgs_expenses),
-        currency,
-        last_month.month,
-        last_month.year,
     )
 
-    return all_orgs_expenses
+    async for org_exp in all_orgs_expenses:
+        yield org_exp
 
 
 async def fetch_unique_billing_currencies(session: AsyncSession) -> Sequence[str]:
@@ -487,61 +461,64 @@ async def update_charges_file_post_generation(
     )
 
 
-async def main(exports_dir: pathlib.Path, settings: Settings) -> None:
+async def genenerate_monthly_charges(
+    session: AsyncSession,
+    currency: str,
+    account: Account,
+    currency_converter: CurrencyConverter,
+    exports_dir: pathlib.Path,
+) -> None:
     today = datetime.now(UTC).date()
+    loop = asyncio.get_event_loop()
 
-    async with session_factory() as session:
-        async with session.begin():
-            unique_billing_currencies = await fetch_unique_billing_currencies(session)
-            currency_converter = await CurrencyConverter.from_db(session)
-            accounts = await fetch_accounts(session)
+    async with session.begin():
+        generated_charges_file = await fetch_existing_generated_charges_file(
+            session, account, currency
+        )
 
-        for currency in unique_billing_currencies:
-            for account in accounts:
-                charges_file_generator = ChargesFileGenerator(
-                    account, currency, currency_converter, exports_dir
-                )
+        if generated_charges_file is not None:
+            return
 
-                async with session.begin():
-                    generated_charges_file = await fetch_existing_generated_charges_file(
-                        session, account, currency
-                    )
+        generator = ChargesFileGenerator(account, currency, currency_converter, exports_dir)
+        async for ds_exp in fetch_datasource_expenses(session, currency):
+            await loop.run_in_executor(None, generator.add_datasource_expense, ds_exp)
 
-                if generated_charges_file is not None:
-                    continue
+        if not generator.has_entries:
+            return
 
-                async with session.begin():
-                    datasource_expenses = await fetch_datasource_expenses(
-                        session, account, currency
-                    )
+        charges_file_db_record = await get_or_create_draft_charges_file(
+            session, account, currency, today
+        )
 
-                df = charges_file_generator.generate_charges_file_dataframe(datasource_expenses)
+    zip_file_path = await loop.run_in_executor(
+        None, generator.save, f"{charges_file_db_record.id}.zip"
+    )
 
-                if df is None:
-                    continue
+    successful_upload = await upload_charges_file_to_azure(charges_file_db_record, zip_file_path)
 
-                async with session.begin():
-                    charges_file_db_record = await get_or_create_draft_charges_file(
-                        session, account, currency, today
-                    )
+    if not successful_upload:  # pragma: no cover
+        return
 
-                exported_filepath = charges_file_generator.export_to_zip(
-                    df, filename=f"{charges_file_db_record.id}.zip"
-                )
+    async with session.begin():
+        await update_charges_file_post_generation(
+            session,
+            charges_file_db_record,
+            amount=generator.running_total.quantize(Decimal("0.01")),
+        )
 
-                successful_upload = await upload_charges_file_to_azure(
-                    charges_file_db_record, exported_filepath
-                )
 
-                if not successful_upload:  # pragma: no cover
-                    continue
+async def main(exports_dir: pathlib.Path, settings: Settings) -> None:
+    async with session_factory() as session:
+        async with session.begin():
+            unique_billing_currencies = await fetch_unique_billing_currencies(session)
+            currency_converter = await CurrencyConverter.from_db(session)
+            accounts = await fetch_accounts(session)
 
-                async with session.begin():
-                    await update_charges_file_post_generation(
-                        session,
-                        charges_file_db_record,
-                        amount=charges_file_generator.get_total_amount(df),
-                    )
+        for currency in unique_billing_currencies:
+            for account in accounts:
+                await genenerate_monthly_charges(
+                    session, currency, account, currency_converter, exports_dir
+                )
 
 
 def command(
diff --git a/app/db/models.py b/app/db/models.py
index 1ca585af..81dfaf9d 100644
--- a/app/db/models.py
+++ b/app/db/models.py
@@ -310,7 +310,7 @@ class DatasourceExpense(Base, HumanReadablePKMixin, TimestampMixin):
         foreign_keys=lambda: [DatasourceExpense.datasource_id],
         viewonly=True,
         uselist=True,
-        lazy="joined",
+        lazy="selectin",
     )
 
     __table_args__ = (
diff --git a/tests/commands/snapshots/test_generate_monthtly_charges/test_export_to_excel/charges.xlsx b/tests/commands/snapshots/test_generate_monthtly_charges/test_export_to_excel/charges.xlsx
deleted file mode 100644
index 99345067..00000000
Binary files a/tests/commands/snapshots/test_generate_monthtly_charges/test_export_to_excel/charges.xlsx and /dev/null differ
diff --git a/tests/commands/snapshots/test_generate_monthtly_charges/test_export_to_zip/charges.csv b/tests/commands/snapshots/test_generate_monthtly_charges/test_export_to_zip/charges.csv
new file mode 100644
index 00000000..81dfbc4f
--- /dev/null
+++ b/tests/commands/snapshots/test_generate_monthtly_charges/test_export_to_zip/charges.csv
@@ -0,0 +1,5 @@
+Entry ID,Subscription Search Criteria,Subscription Search Value,Item Search Criteria,Item Search Value,Usage Start Time,Usage End Time,Quantity,Purchase Price,Total Purchase Price,External Reference,Vendor Description 1,Vendor Description 2,Vendor Reference
+1,subscription.externalIds.vendor,FORG-7330-1645-9608,item.externalIds.vendor,FIN-0001-P1M,1-Mar-2025,31-Mar-2025,1,0.56,0.56,06cb0fb3-9a1d-4644-815e-f6d13b8faa18,AWS Account,a65ed389-b74d-4fb1-b2e7-06298fadc1a6,
+2,subscription.externalIds.vendor,FORG-7330-1645-9608,item.externalIds.vendor,FIN-0001-P1M,1-Mar-2025,31-Mar-2025,1,-0.56,-0.56,06cb0fb3-9a1d-4644-815e-f6d13b8faa18,AWS Account,a65ed389-b74d-4fb1-b2e7-06298fadc1a6,
+3,subscription.externalIds.vendor,FORG-7330-1645-9608,item.externalIds.vendor,FIN-0001-P1M,1-Mar-2025,31-Mar-2025,1,0.65,0.65,06cb0fb3-9a1d-4644-815e-f6d13b8faa18,GCP Account,386ecbe0-6b65-46a4-8b81-48f6b38a088c,
+4,subscription.externalIds.vendor,FORG-7330-1645-9608,item.externalIds.vendor,FIN-0001-P1M,1-Mar-2025,31-Mar-2025,1,-0.65,-0.65,06cb0fb3-9a1d-4644-815e-f6d13b8faa18,GCP Account,386ecbe0-6b65-46a4-8b81-48f6b38a088c,
diff --git a/tests/commands/snapshots/test_generate_monthtly_charges/test_export_to_zip/charges.xlsx b/tests/commands/snapshots/test_generate_monthtly_charges/test_export_to_zip/charges.xlsx
deleted file mode 100644
index 710d4fd6..00000000
Binary files a/tests/commands/snapshots/test_generate_monthtly_charges/test_export_to_zip/charges.xlsx and /dev/null differ
diff --git a/tests/commands/snapshots/test_generate_monthtly_charges/test_export_to_zip/exchange_rates.json b/tests/commands/snapshots/test_generate_monthtly_charges/test_export_to_zip/exchange_rates.json
index a11b83c3..b4d44e04 100644
--- a/tests/commands/snapshots/test_generate_monthtly_charges/test_export_to_zip/exchange_rates.json
+++ b/tests/commands/snapshots/test_generate_monthtly_charges/test_export_to_zip/exchange_rates.json
@@ -1,10 +1,10 @@
 {
     "result": "success",
-    "base_code": "EUR",
+    "base_code": "USD",
     "conversion_rates": {
-        "EUR": 1.0,
-        "GBP": 0.8555,
-        "USD": 1.0808
+        "EUR": 0.9252,
+        "GBP": 0.7737,
+        "USD": 1.0
     },
     "time_last_update_unix": 1744279200,
     "time_next_update_unix": 1744365600
diff --git a/tests/commands/test_generate_monthtly_charges.py b/tests/commands/test_generate_monthtly_charges.py
index a68dab7c..41b37a26 100644
--- a/tests/commands/test_generate_monthtly_charges.py
+++ b/tests/commands/test_generate_monthtly_charges.py
@@ -2,7 +2,7 @@
 import logging
 import pathlib
 import zipfile
-from datetime import UTC, date, datetime
+from datetime import UTC, date, datetime, timedelta
 from decimal import Decimal
 
 import pandas as pd
@@ -202,6 +202,130 @@ async def eur_org_billed_in_gbp_expenses(
     return org
 
 
+@pytest.fixture
+async def gbp_org_billed_in_eur_expenses(
+    organization_factory: ModelFactory[Organization],
+    datasource_expense_factory: ModelFactory[DatasourceExpense],
+    entitlement_factory: ModelFactory[Entitlement],
+    affiliate_account: Account,
+    db_session: AsyncSession,
+    faker: Faker,
+):
+    org = await organization_factory(
+        operations_external_id="ORG-22222",
+        name=faker.company(),
+        currency="GBP",
+        billing_currency="EUR",
+        status=OrganizationStatus.ACTIVE,
+        linked_organization_id=faker.uuid4(str),
+    )
+
+    aws_acc_mar_expenses = await datasource_expense_factory(  # noqa: F841
+        datasource_id=faker.uuid4(str),
+        datasource_name="AWS Account",
+        organization=org,
+        month=3,
+        year=2025,
+        month_expenses=Decimal("50.00"),
+        created_at=datetime(2025, 3, 1, 10, 0, 0, tzinfo=UTC),
+        updated_at=datetime(2025, 3, 31, 10, 0, 0, tzinfo=UTC),
+    )
+
+    azure_acc_mar_expenses = await datasource_expense_factory(  # noqa: F841
+        datasource_id=faker.uuid4(str),
+        datasource_name="Azure Account",
+        organization=org,
+        month=3,
+        year=2025,
+        month_expenses=Decimal("40.00"),
+        created_at=datetime(2025, 3, 1, 10, 0, 0, tzinfo=UTC),
+        updated_at=datetime(2025, 3, 31, 10, 0, 0, tzinfo=UTC),
+    )
+
+    await entitlement_factory(
+        name="Free AWS account",
+        affiliate_external_id="ACC-111111",
+        datasource_id=aws_acc_mar_expenses.datasource_id,
+        status=EntitlementStatus.TERMINATED,  # should be ignored
+        owner=affiliate_account,
+    )
+
+    await entitlement_factory(
+        name="Free Azure account",
+        affiliate_external_id="ACC-888888",
+        datasource_id=azure_acc_mar_expenses.datasource_id,
+        status=EntitlementStatus.ACTIVE,
+        owner=affiliate_account,
+    )
+
+    await db_session.refresh(aws_acc_mar_expenses)
+    await db_session.refresh(azure_acc_mar_expenses)
+
+    return org
+
+
+@pytest.fixture
+async def eur_org_billed_in_eur_expenses(
+    organization_factory: ModelFactory[Organization],
+    datasource_expense_factory: ModelFactory[DatasourceExpense],
+    entitlement_factory: ModelFactory[Entitlement],
+    affiliate_account: Account,
+    db_session: AsyncSession,
+    faker: Faker,
+):
+    org = await organization_factory(
+        operations_external_id="ORG-77777",
+        name=faker.company(),
+        currency="EUR",
+        billing_currency="EUR",
+        status=OrganizationStatus.ACTIVE,
+        linked_organization_id=faker.uuid4(str),
+    )
+
+    aws_acc_mar_expenses = await datasource_expense_factory(  # noqa: F841
+        datasource_id=faker.uuid4(str),
+        datasource_name="AWS Account",
+        organization=org,
+        month=3,
+        year=2025,
+        month_expenses=Decimal("50.00"),
+        created_at=datetime(2025, 3, 1, 10, 0, 0, tzinfo=UTC),
+        updated_at=datetime(2025, 3, 31, 10, 0, 0, tzinfo=UTC),
+    )
+
+    azure_acc_mar_expenses = await datasource_expense_factory(  # noqa: F841
+        datasource_id=faker.uuid4(str),
+        datasource_name="Azure Account",
+        organization=org,
+        month=3,
+        year=2025,
+        month_expenses=Decimal("40.00"),
+        created_at=datetime(2025, 3, 1, 10, 0, 0, tzinfo=UTC),
+        updated_at=datetime(2025, 3, 31, 10, 0, 0, tzinfo=UTC),
+    )
+
+    await entitlement_factory(
+        name="Free AWS account",
+        affiliate_external_id="ACC-99999",
+        datasource_id=aws_acc_mar_expenses.datasource_id,
+        status=EntitlementStatus.TERMINATED,  # should be ignored
+        owner=affiliate_account,
+    )
+
+    await entitlement_factory(
+        name="Free Azure account",
+        affiliate_external_id="ACC-88888",
+        datasource_id=azure_acc_mar_expenses.datasource_id,
+        status=EntitlementStatus.ACTIVE,
+        owner=affiliate_account,
+    )
+
+    await db_session.refresh(aws_acc_mar_expenses)
+    await db_session.refresh(azure_acc_mar_expenses)
+
+    return org
+
+
 @pytest.fixture
 async def usd_org_billed_in_usd_expenses(
     organization_factory: ModelFactory[Organization],
@@ -268,17 +392,118 @@ async def usd_org_billed_in_usd_expenses(
     )
 
 
-def assert_df_matches_snapshot(df: pd.DataFrame, snapshot: Snapshot) -> None:
-    # We're exporting the dataframe to CSV format, so that we can save it in a snapshot file.
+def assert_generated_file_matches_snapshot(
+    excel_file_path: pathlib.Path,
+    snapshot: Snapshot,
+) -> None:
+    # We're exporting the data to CSV format, so that we can save it in a snapshot file.
     # We're intentionally using CSV instead of Excel format, as .xlsx is a binary format and
     # should the test fail, the diff would be unreadable and won't be obvious what's wrong.
-    # There is a seperate test specifically for the excel exporting
+    # Since we're still parsing the file from xlsx that is enough to guarantee that the file
+    # is a valid xlsx file
+
+    csv_file = io.StringIO()
+    df = pd.read_excel(excel_file_path)
+    df.to_csv(csv_file, index=False, header=True, float_format="%.2f")
+
+    csv_file.seek(0)
+    snapshot.assert_match(csv_file.read(), "charges.csv")
+
+
+@time_machine.travel("2025-04-10T10:00:00Z", tick=False)
+def test_charges_file_generator_append_row(
+    currency_converter: CurrencyConverter,
+    operations_account: Account,
+    test_settings: Settings,
+    tmp_path: pathlib.Path,
+):
+    generator = ChargesFileGenerator(operations_account, "USD", currency_converter, tmp_path)
+
+    assert not generator.has_entries
+    assert generator.total_rows == 0
+    assert generator.running_total == Decimal("0.00")
+
+    generator.append_row(
+        ChargeEntry(
+            subscription_search_criteria="subscription.externalIds.vendor",
+            subscription_search_value=operations_account.id,
+            item_search_criteria="item.externalIds.vendor",
+            item_search_value=test_settings.ffc_external_product_id,
+            usage_start_time=(datetime.now(UTC) - timedelta(days=28)).date(),
+            usage_end_time=datetime.now(UTC).date(),
+            price=Decimal("1.00"),
+            external_reference="ORG-1234-5678",
+            vendor_description_1="AWS Account",
+            vendor_description_2="123456789",
+            vendor_reference="",
+        )
+    )
+
+    assert generator.has_entries
+    assert generator.total_rows == 2  # with the header
+    assert generator.running_total == Decimal("1.00")
+
+    generator.append_row(
+        ChargeEntry(
+            subscription_search_criteria="subscription.externalIds.vendor",
+            subscription_search_value=operations_account.id,
+            item_search_criteria="item.externalIds.vendor",
+            item_search_value=test_settings.ffc_external_product_id,
+            usage_start_time=(datetime.now(UTC) - timedelta(days=28)).date(),
+            usage_end_time=datetime.now(UTC).date(),
+            price=Decimal("2.00"),
+            external_reference="ORG-1234-5678",
+            vendor_description_1="GCP Account",
+            vendor_description_2="987654321",
+            vendor_reference="",
+        )
+    )
+
+    assert generator.has_entries
+    assert generator.total_rows == 3  # with the header
+    assert generator.running_total == Decimal("3.00")
+
+
+@pytest.mark.parametrize(
+    ("account_fixture", "currency", "expected_total_rows", "expected_running_total"),
+    [
+        ("operations_account", "EUR", 5, Decimal("0.00")),
+        ("affiliate_account", "EUR", 2, Decimal("0.65")),
+        ("another_affiliate_account", "EUR", 2, Decimal("0.56")),
+        ("operations_account", "GBP", 4, Decimal("0.43")),
+        ("affiliate_account", "GBP", 2, Decimal("0.34")),
+        ("another_affiliate_account", "GBP", 0, Decimal("0.00")),
+        ("operations_account", "USD", 5, Decimal("1.10")),
+        ("affiliate_account", "USD", 2, Decimal("0.60")),
+        ("another_affiliate_account", "USD", 0, Decimal("0.00")),
+    ],
+)
+@time_machine.travel("2025-04-10T10:00:00Z", tick=False)
+async def test_charges_file_generator_add_datasource_expense(
+    currency_converter: CurrencyConverter,
+    usd_org_billed_in_usd_expenses: Organization,
+    usd_org_billed_in_eur_expenses: Organization,
+    eur_org_billed_in_gbp_expenses: Organization,
+    test_settings: Settings,
+    db_session: AsyncSession,
+    request: pytest.FixtureRequest,
+    tmp_path: pathlib.Path,
+    account_fixture: str,
+    currency: str,
+    expected_total_rows: int,
+    expected_running_total: Decimal,
+):
+    account = request.getfixturevalue(account_fixture)
+
+    generator = ChargesFileGenerator(account, currency, currency_converter, tmp_path)
 
-    file = io.StringIO()
-    df.to_csv(file, index=False, header=True)
+    async for ds_exp in fetch_datasource_expenses(db_session, currency):
+        generator.add_datasource_expense(ds_exp)
 
-    file.seek(0)
-    snapshot.assert_match(file.read(), "charges.csv")
+    generator.save("charges.xlsx")
+    assert generator.has_entries == (generator.total_rows > 0)
+    assert generator.total_rows == expected_total_rows  # with the header
+    assert generator.running_total.quantize(Decimal("0.00")) == expected_running_total
 
 
 @pytest.mark.fixed_random_seed
@@ -291,12 +516,12 @@ async def test_generate_charges_file_dataframe_operations_same_currency(
     db_session: AsyncSession,
     tmp_path: pathlib.Path,
 ):
-    charges_file_generator = ChargesFileGenerator(
-        operations_account, "USD", currency_converter, tmp_path
-    )
-    datasource_expenses = await fetch_datasource_expenses(db_session, operations_account, "USD")
-    df = charges_file_generator.generate_charges_file_dataframe(datasource_expenses)
-    assert_df_matches_snapshot(df, snapshot)
+    generator = ChargesFileGenerator(operations_account, "USD", currency_converter, tmp_path)
+
+    async for ds_exp in fetch_datasource_expenses(db_session, "USD"):
+        generator.add_datasource_expense(ds_exp)
+
+    assert_generated_file_matches_snapshot(generator.save("charges.xlsx"), snapshot)
 
 
 @pytest.mark.fixed_random_seed
@@ -309,12 +534,12 @@ async def test_generate_charges_file_dataframe_affiliate_same_currency(
     snapshot: Snapshot,
     tmp_path: pathlib.Path,
 ):
-    charges_file_generator = ChargesFileGenerator(
-        affiliate_account, "USD", currency_converter, tmp_path
-    )
-    datasource_expenses = await fetch_datasource_expenses(db_session, affiliate_account, "USD")
-    df = charges_file_generator.generate_charges_file_dataframe(datasource_expenses)
-    assert_df_matches_snapshot(df, snapshot)
+    generator = ChargesFileGenerator(affiliate_account, "USD", currency_converter, tmp_path)
+
+    async for ds_exp in fetch_datasource_expenses(db_session, "USD"):
+        generator.add_datasource_expense(ds_exp)
+
+    assert_generated_file_matches_snapshot(generator.save("charges.xlsx"), snapshot)
 
 
 async def test_generate_charges_file_dataframe_empty_file(
@@ -324,12 +549,12 @@ async def test_generate_charges_file_dataframe_empty_file(
     db_session: AsyncSession,
     tmp_path: pathlib.Path,
 ):
-    charges_file_generator = ChargesFileGenerator(
-        affiliate_account, "EUR", currency_converter, tmp_path
-    )
-    datasource_expenses = await fetch_datasource_expenses(db_session, affiliate_account, "EUR")
-    df = charges_file_generator.generate_charges_file_dataframe(datasource_expenses)
-    assert df is None
+    generator = ChargesFileGenerator(affiliate_account, "EUR", currency_converter, tmp_path)
+
+    async for ds_exp in fetch_datasource_expenses(db_session, "EUR"):
+        generator.add_datasource_expense(ds_exp)
+
+    assert not generator.has_entries
 
 
 @pytest.mark.parametrize(
@@ -349,67 +574,69 @@ async def test_generate_charges_file_dataframe_affiliate_multiple_organizations_
     tmp_path: pathlib.Path,
     currency: str,
 ):
-    charges_file_generator = ChargesFileGenerator(
-        affiliate_account, currency, currency_converter, tmp_path
-    )
-    datasource_expenses = await fetch_datasource_expenses(db_session, affiliate_account, currency)
-    df = charges_file_generator.generate_charges_file_dataframe(datasource_expenses)
-    assert_df_matches_snapshot(df, snapshot)
+    generator = ChargesFileGenerator(affiliate_account, currency, currency_converter, tmp_path)
+
+    async for ds_exp in fetch_datasource_expenses(db_session, currency):
+        generator.add_datasource_expense(ds_exp)
+
+    assert_generated_file_matches_snapshot(generator.save("charges.xlsx"), snapshot)
 
 
 @pytest.mark.fixed_random_seed
 @time_machine.travel("2025-04-10T10:00:00Z", tick=False)
-async def test_export_to_excel(
-    currency_converter: CurrencyConverter,
+async def test_export_to_zip(
+    exchange_rates_factory: ModelFactory[ExchangeRates],
     operations_account: Account,
-    usd_org_billed_in_usd_expenses: Organization,
+    usd_org_billed_in_eur_expenses: Organization,
+    currency_converter: CurrencyConverter,
     db_session: AsyncSession,
     snapshot: Snapshot,
     tmp_path: pathlib.Path,
 ):
-    charges_file_generator = ChargesFileGenerator(
-        operations_account, "USD", currency_converter, tmp_path
-    )
-    datasource_expenses = await fetch_datasource_expenses(db_session, operations_account, "USD")
-    df = charges_file_generator.generate_charges_file_dataframe(datasource_expenses)
-    assert df is not None
+    generator = ChargesFileGenerator(operations_account, "EUR", currency_converter, tmp_path)
 
-    filepath = charges_file_generator.export_to_excel(df, "charges.xlsx")
+    async for ds_exp in fetch_datasource_expenses(db_session, "EUR"):
+        generator.add_datasource_expense(ds_exp)
 
-    assert filepath.name == "charges.xlsx"
-    assert filepath.parent == tmp_path
-    snapshot.assert_match(filepath.read_bytes(), "charges.xlsx")
+    assert generator.has_entries
+
+    zip_file_path = generator.make_archive("charges.zip")
+
+    with zipfile.ZipFile(zip_file_path, "r") as archive:
+        assert sorted(archive.namelist()) == ["charges.xlsx", "exchange_rates_USD.json"]
+
+        excel_file_path = pathlib.Path(archive.extract("charges.xlsx", tmp_path))
+        assert_generated_file_matches_snapshot(excel_file_path, snapshot)
+        snapshot.assert_match(archive.read("exchange_rates_USD.json"), "exchange_rates.json")
 
 
-@pytest.mark.fixed_random_seed
 @time_machine.travel("2025-04-10T10:00:00Z", tick=False)
-async def test_export_to_zip(
+async def test_export_to_zip_includes_multiple_exchange_rates(
     exchange_rates_factory: ModelFactory[ExchangeRates],
     operations_account: Account,
     usd_org_billed_in_eur_expenses: Organization,
+    eur_org_billed_in_eur_expenses: Organization,
+    gbp_org_billed_in_eur_expenses: Organization,
     currency_converter: CurrencyConverter,
     db_session: AsyncSession,
     snapshot: Snapshot,
     tmp_path: pathlib.Path,
 ):
-    charges_file_generator = ChargesFileGenerator(
-        operations_account, "EUR", currency_converter, tmp_path
-    )
-
-    datasource_expenses = await fetch_datasource_expenses(db_session, operations_account, "EUR")
-    df = charges_file_generator.generate_charges_file_dataframe(datasource_expenses)
-    assert df is not None
+    generator = ChargesFileGenerator(operations_account, "EUR", currency_converter, tmp_path)
 
-    filepath = charges_file_generator.export_to_zip(df, filename="charges.zip")
+    async for ds_exp in fetch_datasource_expenses(db_session, "EUR"):
+        generator.add_datasource_expense(ds_exp)
 
-    assert filepath.name == "charges.zip"
-    assert filepath.parent == tmp_path
+    assert generator.has_entries
 
-    with zipfile.ZipFile(filepath, "r") as archive:
-        assert sorted(archive.namelist()) == ["charges.xlsx", "exchange_rates_EUR.json"]
+    zip_file_path = generator.make_archive("charges.zip")
 
-        snapshot.assert_match(archive.read("charges.xlsx"), "charges.xlsx")
-        snapshot.assert_match(archive.read("exchange_rates_EUR.json"), "exchange_rates.json")
+    with zipfile.ZipFile(zip_file_path, "r") as archive:
+        assert sorted(archive.namelist()) == [
+            "charges.xlsx",
+            "exchange_rates_GBP.json",
+            "exchange_rates_USD.json",
+        ]
 
 
 @pytest.mark.parametrize(
@@ -428,25 +655,24 @@ async def test_get_total_amount(
     usd_org_billed_in_usd_expenses: Organization,
     usd_org_billed_in_eur_expenses: Organization,
     eur_org_billed_in_gbp_expenses: Organization,
-    snapshot: Snapshot,
     tmp_path: pathlib.Path,
+    snapshot: Snapshot,
     db_session: AsyncSession,
     currency: str,
     expected_total_amount: Decimal,
 ):
-    charges_file_generator = ChargesFileGenerator(
-        operations_account, currency, currency_converter, tmp_path
-    )
-    datasource_expenses = await fetch_datasource_expenses(db_session, operations_account, currency)
-    df = charges_file_generator.generate_charges_file_dataframe(datasource_expenses)
+    generator = ChargesFileGenerator(operations_account, currency, currency_converter, tmp_path)
+    async for ds_exp in fetch_datasource_expenses(db_session, currency):
+        generator.add_datasource_expense(ds_exp)
 
-    assert df is not None
+    assert generator.has_entries
 
-    total_amount = charges_file_generator.get_total_amount(df)
+    df = pd.read_excel(generator.save("charges.xlsx"))
 
+    total_amount = generator.running_total.quantize(Decimal("0.00"))
     assert total_amount == expected_total_amount
-    assert total_amount == df["Purchase Price"].sum()  # type: ignore[index]
-    assert total_amount == df["Total Purchase Price"].sum()  # type: ignore[index]
+    assert total_amount == Decimal(df["Purchase Price"].sum()).quantize(Decimal("0.00"))
+    assert total_amount == Decimal(df["Total Purchase Price"].sum()).quantize(Decimal("0.00"))
 
 
 @pytest.mark.parametrize(
@@ -765,7 +991,7 @@ async def test_full_run(
     # (both new records and updating existing ones), so that we can filter the affected records
     # bellow
     with caplog.at_level(logging.INFO), time_machine.travel("2025-04-10T11:00:00Z", tick=False):
-        await generate_monthly_charges_main(tmp_path, test_settings)
+        await generate_monthly_charges_main(exports_dir=tmp_path, settings=test_settings)
 
     await db_session.refresh(generated_charges_file)
     await db_session.refresh(processed_charges_file)
@@ -831,22 +1057,9 @@ async def test_full_run(
     )
     assert "Found 4 accounts in the database" in found_logs
 
-    assert "Found 1 organizations to process for billing currency EUR" in found_logs
-    assert "Found 1 organizations to process for billing currency GBP" in found_logs
-    assert "Found 1 organizations to process for billing currency USD" in found_logs
-
-    assert (
-        "Found 2 datasource expenses for all organizations "
-        "with EUR billing currency for month = 3, year = 2025"
-    ) in found_logs
-    assert (
-        "Found 2 datasource expenses for all organizations "
-        "with GBP billing currency for month = 3, year = 2025"
-    ) in found_logs
-    assert (
-        "Found 3 datasource expenses for all organizations "
-        "with USD billing currency for month = 3, year = 2025"
-    ) in found_logs
+    assert "Found 1 organizations to process with billing currency EUR" in found_logs
+    assert "Found 1 organizations to process with billing currency GBP" in found_logs
+    assert "Found 1 organizations to process with billing currency USD" in found_logs
 
     # filtering "Charges file ..." logs, so that tests failures are easier to debug
     charges_file_logs = [msg for msg in caplog.messages if msg.startswith("Charges file ")]


Reviews:
user: login: ffaraone
state: CHANGES_REQUESTED
author_association: COLLABORATOR
submitted_at: 2025-05-02T11:20:19Z
user: login: arturbalabanov
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-05-02T11:36:42Z
user: login: arturbalabanov
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-05-02T11:42:19Z
user: login: arturbalabanov
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-05-02T11:48:10Z
user: login: arturbalabanov
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-05-02T12:28:33Z
user: login: ffaraone
state: APPROVED
author_association: COLLABORATOR
submitted_at: 2025-05-02T15:00:06Z
user: login: arturbalabanov
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-05-02T16:55:11Z

Review comments:
diff_hunk: @@ -95,18 +95,43 @@ class ChargesFileGenerator:
     currency_converter: CurrencyConverter
     exports_dir: pathlib.Path
 
-    def get_charge_entries(
-        self, datasource_expenses: Sequence[DatasourceExpense]
-    ) -> list[ChargeEntry]:
-        return [
-            charge_entry
-            for ds_exp in datasource_expenses
-            for charge_entry in self.charge_entries_for_datasource_expense(ds_exp)
-        ]
-
-    def charge_entries_for_datasource_expense(
-        self, datasource_expense: DatasourceExpense
-    ) -> list[ChargeEntry]:
+    def __post_init__(self) -> None:
path: app/commands/generate_monthly_charges.py
user: login: ffaraone
body: I would strongly suggest to convert this from a dataclass to a class. 
Dataclasses are intended for when you primarily need to store data.
created_at: 2025-05-02T11:17:10Z
author_association: COLLABORATOR
diff_hunk: @@ -498,39 +455,31 @@ async def main(exports_dir: pathlib.Path, settings: Settings) -> None:
 
         for currency in unique_billing_currencies:
             for account in accounts:
-                charges_file_generator = ChargesFileGenerator(
-                    account, currency, currency_converter, exports_dir
-                )
-
                 async with session.begin():
                     generated_charges_file = await fetch_existing_generated_charges_file(
                         session, account, currency
                     )
 
-                if generated_charges_file is not None:
-                    continue
+                    if generated_charges_file is not None:
+                        continue
 
-                async with session.begin():
-                    datasource_expenses = await fetch_datasource_expenses(
-                        session, account, currency
+                    generator = ChargesFileGenerator(
+                        account, currency, currency_converter, exports_dir
                     )
+                    async for ds_exp in fetch_datasource_expenses(session, currency):
path: app/commands/generate_monthly_charges.py
user: login: ffaraone
body: This fetch should be moved outside the loop of accounts. otherwise you will fetch the expenses for the same currency N times where N is the amount of accounts.
created_at: 2025-05-02T11:19:27Z
author_association: COLLABORATOR
diff_hunk: @@ -498,39 +455,31 @@ async def main(exports_dir: pathlib.Path, settings: Settings) -> None:
 
         for currency in unique_billing_currencies:
             for account in accounts:
-                charges_file_generator = ChargesFileGenerator(
-                    account, currency, currency_converter, exports_dir
-                )
-
                 async with session.begin():
                     generated_charges_file = await fetch_existing_generated_charges_file(
                         session, account, currency
                     )
 
-                if generated_charges_file is not None:
-                    continue
+                    if generated_charges_file is not None:
+                        continue
 
-                async with session.begin():
-                    datasource_expenses = await fetch_datasource_expenses(
-                        session, account, currency
+                    generator = ChargesFileGenerator(
+                        account, currency, currency_converter, exports_dir
                     )
+                    async for ds_exp in fetch_datasource_expenses(session, currency):
+                        generator.add_datasource_expense(ds_exp)
 
-                df = charges_file_generator.generate_charges_file_dataframe(datasource_expenses)
-
-                if df is None:
-                    continue
+                    if not generator.has_entries:
+                        continue
 
-                async with session.begin():
                     charges_file_db_record = await get_or_create_draft_charges_file(
                         session, account, currency, today
                     )
 
-                exported_filepath = charges_file_generator.export_to_zip(
-                    df, filename=f"{charges_file_db_record.id}.zip"
-                )
+                zip_file_path = generator.make_archive(f"{charges_file_db_record.id}.zip")
path: app/commands/generate_monthly_charges.py
user: login: ffaraone
body: This should be run into a thread otherwise you will block the event loop.
created_at: 2025-05-02T11:19:46Z
author_association: COLLABORATOR
diff_hunk: @@ -95,18 +95,43 @@ class ChargesFileGenerator:
     currency_converter: CurrencyConverter
     exports_dir: pathlib.Path
 
-    def get_charge_entries(
-        self, datasource_expenses: Sequence[DatasourceExpense]
-    ) -> list[ChargeEntry]:
-        return [
-            charge_entry
-            for ds_exp in datasource_expenses
-            for charge_entry in self.charge_entries_for_datasource_expense(ds_exp)
-        ]
-
-    def charge_entries_for_datasource_expense(
-        self, datasource_expense: DatasourceExpense
-    ) -> list[ChargeEntry]:
+    def __post_init__(self) -> None:
path: app/commands/generate_monthly_charges.py
user: login: arturbalabanov
body: Good point! Done
created_at: 2025-05-02T11:36:41Z
author_association: CONTRIBUTOR
diff_hunk: @@ -498,39 +455,31 @@ async def main(exports_dir: pathlib.Path, settings: Settings) -> None:
 
         for currency in unique_billing_currencies:
             for account in accounts:
-                charges_file_generator = ChargesFileGenerator(
-                    account, currency, currency_converter, exports_dir
-                )
-
                 async with session.begin():
                     generated_charges_file = await fetch_existing_generated_charges_file(
                         session, account, currency
                     )
 
-                if generated_charges_file is not None:
-                    continue
+                    if generated_charges_file is not None:
+                        continue
 
-                async with session.begin():
-                    datasource_expenses = await fetch_datasource_expenses(
-                        session, account, currency
+                    generator = ChargesFileGenerator(
+                        account, currency, currency_converter, exports_dir
                     )
+                    async for ds_exp in fetch_datasource_expenses(session, currency):
+                        generator.add_datasource_expense(ds_exp)
 
-                df = charges_file_generator.generate_charges_file_dataframe(datasource_expenses)
-
-                if df is None:
-                    continue
+                    if not generator.has_entries:
+                        continue
 
-                async with session.begin():
                     charges_file_db_record = await get_or_create_draft_charges_file(
                         session, account, currency, today
                     )
 
-                exported_filepath = charges_file_generator.export_to_zip(
-                    df, filename=f"{charges_file_db_record.id}.zip"
-                )
+                zip_file_path = generator.make_archive(f"{charges_file_db_record.id}.zip")
path: app/commands/generate_monthly_charges.py
user: login: arturbalabanov
body: Is that an issue in this scenario though? I'm asking because:

1. I _think_ making the archive will be a fairly quick operation anyway, so it may not be a big deal
2. We don't have large performance requirements, so even if some operations are done sequentially it will be fine
3. The file I/O of generating the excel files is synchronous anyway and I don't think `openpyxls` supports `asyncio`. We could use threads for that too but it would be an overkill I think

On the other hand, adding threads, making sure they are in sync, handling exceptions inside them etc. just introduces a lot more complexity. I don't think it's worth it but maybe I'm missing something?
created_at: 2025-05-02T11:42:19Z
author_association: CONTRIBUTOR
diff_hunk: @@ -498,39 +455,31 @@ async def main(exports_dir: pathlib.Path, settings: Settings) -> None:
 
         for currency in unique_billing_currencies:
             for account in accounts:
-                charges_file_generator = ChargesFileGenerator(
-                    account, currency, currency_converter, exports_dir
-                )
-
                 async with session.begin():
                     generated_charges_file = await fetch_existing_generated_charges_file(
                         session, account, currency
                     )
 
-                if generated_charges_file is not None:
-                    continue
+                    if generated_charges_file is not None:
+                        continue
 
-                async with session.begin():
-                    datasource_expenses = await fetch_datasource_expenses(
-                        session, account, currency
+                    generator = ChargesFileGenerator(
+                        account, currency, currency_converter, exports_dir
                     )
+                    async for ds_exp in fetch_datasource_expenses(session, currency):
path: app/commands/generate_monthly_charges.py
user: login: arturbalabanov
body: Yep, initially that's what I tried and it's possible but as I wrote in the description it will increase the code complexity a lot. The main reason for this is that this for loop needs to run in a transaction that's only closed when the loop ends. That means that the archive generation and (maybe most importantly) the file upload to Azure will block the transaction unless a large refactoring is done. The excel file generation is even now part of that transaction but we don't really have a choice and it shouldn't take too long anyway. But that's only for one file.

We could extract these operations after the loop but I honestly don't think it's worth it as the code becomes very hard to follow. At the moment making the script more flexible is fairly easy but it will be harder if we do this. Feels like premature optimisation to me, shall we leave it as it is and go back to it if it becomes a problem?
created_at: 2025-05-02T11:48:10Z
author_association: CONTRIBUTOR
diff_hunk: @@ -222,10 +199,11 @@ def export_to_zip(self, df: pd.DataFrame, filename: str) -> pathlib.Path:
             self.currency,
         )
 
-        excel_filepath = self.export_to_excel(df, filename="charges.xlsx")
+        excel_filepath = self.save("charges.xlsx")
+        filepath = self.exports_dir / filename
 
         with zipfile.ZipFile(filepath, mode="w") as archive:
-            archive.write(excel_filepath, arcname=excel_filepath.name)
+            archive.write(excel_filepath, arcname="charges.xlsx")
             archive.writestr(
                 f"exchange_rates_{self.currency}.json",
                 self.currency_converter.get_exchangerate_api_response_json(self.currency),
path: app/commands/generate_monthly_charges.py
user: login: arturbalabanov
body: Include all the exchange rates currencies **from*** not the **to** currency's exchange rates
created_at: 2025-05-02T12:28:33Z
author_association: CONTRIBUTOR
diff_hunk: @@ -549,6 +530,14 @@ def command(
     exports_dir: Annotated[
         pathlib.Path, typer.Option("--exports-dir", help="Directory to export the charge files to")
     ],
+    max_workers: Annotated[
path: app/commands/generate_monthly_charges.py
user: login: ffaraone
body: There's no need in this case to control the number of workers in the thread pool, especially because your code will always use 1 thread.
If you don't pass a thread pool to the `run_in_executor` it will use the default executor (see: https://docs.python.org/3/library/asyncio-eventloop.html#asyncio.loop.run_in_executor).
created_at: 2025-05-02T14:59:51Z
author_association: COLLABORATOR
diff_hunk: @@ -549,6 +530,14 @@ def command(
     exports_dir: Annotated[
         pathlib.Path, typer.Option("--exports-dir", help="Directory to export the charge files to")
     ],
+    max_workers: Annotated[
path: app/commands/generate_monthly_charges.py
user: login: arturbalabanov
body: Yeah you're right, don't know what I was thinking there :D Changed to the default executor
created_at: 2025-05-02T16:55:10Z
author_association: CONTRIBUTOR

PR comments:
user: login: sonarqubecloud[bot]
created_at: 2025-05-02T12:20:03Z
body: ## [![Quality Gate Passed](https://sonarsource.github.io/sonarcloud-github-static-resources/v2/checks/QualityGateBadge/qg-passed-20px.png 'Quality Gate Passed')](https://sonarcloud.io/dashboard?id=softwareone-platform_ffc-finops-operations&pullRequest=179) **Quality Gate passed**  
Issues  
![](https://sonarsource.github.io/sonarcloud-github-static-resources/v2/common/passed-16px.png '') [4 New issues](https://sonarcloud.io/project/issues?id=softwareone-platform_ffc-finops-operations&pullRequest=179&issueStatuses=OPEN,CONFIRMED&sinceLeakPeriod=true)  
![](https://sonarsource.github.io/sonarcloud-github-static-resources/v2/common/accepted-16px.png '') [0 Accepted issues](https://sonarcloud.io/project/issues?id=softwareone-platform_ffc-finops-operations&pullRequest=179&issueStatuses=ACCEPTED)

Measures  
![](https://sonarsource.github.io/sonarcloud-github-static-resources/v2/common/passed-16px.png '') [0 Security Hotspots](https://sonarcloud.io/project/security_hotspots?id=softwareone-platform_ffc-finops-operations&pullRequest=179&issueStatuses=OPEN,CONFIRMED&sinceLeakPeriod=true)  
![](https://sonarsource.github.io/sonarcloud-github-static-resources/v2/common/passed-16px.png '') [96.4% Coverage on New Code](https://sonarcloud.io/component_measures?id=softwareone-platform_ffc-finops-operations&pullRequest=179&metric=new_coverage&view=list)  
![](https://sonarsource.github.io/sonarcloud-github-static-resources/v2/common/passed-16px.png '') [0.0% Duplication on New Code](https://sonarcloud.io/component_measures?id=softwareone-platform_ffc-finops-operations&pullRequest=179&metric=new_duplicated_lines_density&view=list)  
  
[See analysis details on SonarQube Cloud](https://sonarcloud.io/dashboard?id=softwareone-platform_ffc-finops-operations&pullRequest=179)


author_association: NONE