Repository: CCI-MOC/invoicing
Pull request #159: Implemented processor to fetch Coldfront allocation data
Author: login: QuanMPhm
Created: 2025-03-05T00:15:14Z | Merged: 2025-04-30T13:44:00Z

Closes #153. Authentication to Coldfront must be done through Keycloak by providing the client id and secret. These can be set through env vars (`KEYCLOAK_CLIENT_ID` and `KEYCLOAK_CLIENT_SECRET`). A Keycloak and Coldfront url can optionally be provided. They default to the ones hosted by the MGHPCC.

The processor is responsible for adding the PI email and institute code to invoices.
If a Coldfront allocation for an invoice project could not be found, and this project also happens to be
billable, a warning will be logged.

**The query to the Coldfront allocation API will only ask for allocations that appeared on the invoice**

Test cases have been added.

Linked issue #153: Implement processor to fetch project information from Coldfront
Instead of having coldfront information fetched from our billing scripts earlier in the pipeline, we believe it is more fitting to have this information fetched from the `invoicing` repo instead, contained within a processor class.

This issue is also to allow us to update the way we are currently querying the Coldfront allocations API. We are querying ALL allocations, and this method will not scale well as the NERC grows.
Linked issue comments: user: login: QuanMPhm
created_at: 2025-03-04T19:54:59Z
body: @knikolla @naved001 Should I add a warning if a project in the invoices does not exist on Coldfront?
author_association: CONTRIBUTOR
user: login: naved001
created_at: 2025-03-04T20:02:37Z
body: @QuanMPhm sure, just log a warning.
author_association: COLLABORATOR
user: login: knikolla
created_at: 2025-03-04T23:04:37Z
body: @QuanMPhm Log a warning if a project does not exist in ColdFront AND it is not a non-billable project.

If something does not exist in ColdFront, but we've already gone through the trouble of adding it to non-billable, then that means we do know about it and there's no need to log a warning. If you want to log something, you could perhaps log a debug statement for this case.
author_association: CONTRIBUTOR
user: login: morantara
created_at: 2025-04-07T19:30:43Z
body: @QuanMPhm this has been in review for a month, is there a status update here? 
author_association: NONE
user: login: QuanMPhm
created_at: 2025-04-08T15:27:49Z
body: @morantara I'm waiting for one more review on this [PR](https://github.com/CCI-MOC/invoicing/pull/159)
author_association: CONTRIBUTOR
user: login: morantara
created_at: 2025-04-28T14:30:17Z
body: @QuanMPhm did you get the reviews you needed for this to be closed in this sprint?
author_association: NONE
user: login: QuanMPhm
created_at: 2025-04-28T17:40:25Z
body: @morantara The PR for this issue is approved, however, it requires that 2 PRs from other repos are approved as well. This set of 3 issues is currently waiting on this [PR](https://github.com/CCI-MOC/openstack-billing-from-db/pull/92)
author_association: CONTRIBUTOR

Diff:
diff --git a/process_report/process_report.py b/process_report/process_report.py
index 01f73246..5a838ef4 100644
--- a/process_report/process_report.py
+++ b/process_report/process_report.py
@@ -3,6 +3,7 @@
 import datetime
 import logging
 from decimal import Decimal
+import os
 
 import pandas
 import pyarrow
@@ -20,6 +21,7 @@
     prepay_credits_snapshot,
 )
 from process_report.processors import (
+    coldfront_fetch_processor,
     validate_pi_alias_processor,
     add_institution_processor,
     lenovo_processor,
@@ -61,6 +63,8 @@
 ALIAS_S3_FILEPATH = "PIs/alias.csv"
 PREPAY_DEBITS_S3_FILEPATH = "Prepay/prepay_debits.csv"
 
+REQUIRED_ENV_VARS = ("KEYCLOAK_CLIENT_ID", "KEYCLOAK_CLIENT_SECRET")
+
 
 logger = logging.getLogger(__name__)
 logging.basicConfig(level=logging.INFO)
@@ -96,6 +100,12 @@ def get_iso8601_time():
     return datetime.datetime.now().strftime("%Y%m%dT%H%M%SZ")
 
 
+def validate_required_env_vars():
+    for required_env_var in REQUIRED_ENV_VARS:
+        if required_env_var not in os.environ:
+            sys.exit(f"Required environment variable {required_env_var} is not set")
+
+
 def main():
     """Remove non-billable PIs and projects"""
 
@@ -220,6 +230,8 @@ def main():
     )
     args = parser.parse_args()
 
+    validate_required_env_vars()
+
     invoice_month = args.invoice_month
 
     csv_files = args.csv_files or fetch_s3_invoices(invoice_month)
@@ -258,8 +270,13 @@ def main():
 
     ### Preliminary processing
 
+    coldfront_fetch_proc = coldfront_fetch_processor.ColdfrontFetchProcessor(
+        "", invoice_month, merged_dataframe, projects
+    )
+    coldfront_fetch_proc.process()
+
     validate_pi_alias_proc = validate_pi_alias_processor.ValidatePIAliasProcessor(
-        "", invoice_month, merged_dataframe, alias_dict
+        "", invoice_month, coldfront_fetch_proc.data, alias_dict
     )
     validate_pi_alias_proc.process()
 
diff --git a/process_report/processors/coldfront_fetch_processor.py b/process_report/processors/coldfront_fetch_processor.py
new file mode 100644
index 00000000..a6c73c3b
--- /dev/null
+++ b/process_report/processors/coldfront_fetch_processor.py
@@ -0,0 +1,106 @@
+import os
+import sys
+import functools
+import logging
+from dataclasses import dataclass
+
+import requests
+
+from process_report.invoices import invoice
+from process_report.processors import processor
+
+logger = logging.getLogger(__name__)
+logging.basicConfig(level=logging.INFO)
+
+
+CF_ATTR_ALLOCATED_PROJECT_NAME = "Allocated Project Name"
+
+
+@dataclass
+class ColdfrontFetchProcessor(processor.Processor):
+    nonbillable_projects: list[str]
+
+    @functools.cached_property
+    def coldfront_client(self):
+        keycloak_url = os.environ.get("KEYCLOAK_URL", "https://keycloak.mss.mghpcc.org")
+
+        # Authenticate with Keycloak
+        token_url = f"{keycloak_url}/auth/realms/mss/protocol/openid-connect/token"
+        r = requests.post(
+            token_url,
+            data={"grant_type": "client_credentials"},
+            auth=requests.auth.HTTPBasicAuth(
+                os.environ["KEYCLOAK_CLIENT_ID"],
+                os.environ["KEYCLOAK_CLIENT_SECRET"],
+            ),
+        )
+        try:
+            r.raise_for_status()
+        except requests.HTTPError:
+            sys.exit(f"Keycloak authentication failed:\n{r.status_code} {r.text}")
+
+        client_token = r.json()["access_token"]
+
+        session = requests.session()
+        headers = {
+            "Authorization": f"Bearer {client_token}",
+            "Content-Type": "application/json",
+        }
+        session.headers.update(headers)
+        return session
+
+    def _get_project_list(self):
+        return self.data[invoice.PROJECT_FIELD].unique()
+
+    def _fetch_coldfront_allocation_api(self, allocation_list):
+        coldfront_api_url = os.environ.get(
+            "COLDFRONT_URL", "https://coldfront.mss.mghpcc.org/api/allocations"
+        )
+        api_query_str = "&".join(
+            [
+                f"attr_{CF_ATTR_ALLOCATED_PROJECT_NAME}={project}"
+                for project in allocation_list
+            ]
+        )
+        r = self.coldfront_client.get(f"{coldfront_api_url}?{api_query_str}")
+
+        return r.json()
+
+    def _get_allocation_data(self, coldfront_api_data):
+        allocation_data = {}
+        for project, project_dict in coldfront_api_data.items():
+            allocation_data[project] = {
+                invoice.PI_FIELD: project_dict["project"]["pi"],
+                invoice.INSTITUTION_ID_FIELD: project_dict["attributes"][
+                    "Institution-Specific Code"
+                ],
+            }
+        return allocation_data
+
+    def _validate_allocation_data(self, allocation_data):
+        missing_projects = (
+            set(self._get_project_list())
+            - set(allocation_data.keys())
+            - set(self.nonbillable_projects)
+        )
+        missing_projects = list(missing_projects)
+        missing_projects.sort()  # Ensures order for testing purposes
+        if missing_projects:
+            logger.warning(
+                f"Projects {missing_projects} not found in Coldfront and are billable! Please check the project names"
+            )
+
+    def _apply_allocation_data(self, allocation_data):
+        for project, data in allocation_data.items():
+            mask = self.data[invoice.PROJECT_FIELD] == project
+            self.data.loc[mask, invoice.PI_FIELD] = data[invoice.PI_FIELD]
+            self.data.loc[mask, invoice.INSTITUTION_ID_FIELD] = data[
+                invoice.INSTITUTION_ID_FIELD
+            ]
+
+    def _process(self):
+        project_allocations_list = self._get_project_list()
+        api_data = self._fetch_coldfront_allocation_api(project_allocations_list)
+        allocation_data = self._get_allocation_data(api_data)
+        self._validate_allocation_data(allocation_data)
+        self._apply_allocation_data(allocation_data)
diff --git a/process_report/tests/unit/processors/test_coldfront_fetch_processor.py b/process_report/tests/unit/processors/test_coldfront_fetch_processor.py
new file mode 100644
index 00000000..b9183ff4
--- /dev/null
+++ b/process_report/tests/unit/processors/test_coldfront_fetch_processor.py
@@ -0,0 +1,92 @@
+from unittest import TestCase, mock
+import pandas
+
+from process_report.tests import util as test_utils
+
+
+class TestColdfrontFetchProcessor(TestCase):
+    def _get_test_invoice(
+        self,
+        project,
+        pi=None,
+        institute_code=None,
+    ):
+        if not pi:
+            pi = [""] * len(project)
+
+        if not institute_code:
+            institute_code = [""] * len(project)
+
+        return pandas.DataFrame(
+            {
+                "Manager (PI)": pi,
+                "Project - Allocation": project,
+                "Institution - Specific Code": institute_code,
+            }
+        )
+
+    def _get_mock_allocation_data(self, project_list, pi_list, institute_code_list):
+        mock_data = {}
+        for i, project in enumerate(project_list):
+            mock_data[project] = {}
+            mock_data[project]["project"] = {"pi": pi_list[i]}
+            mock_data[project]["attributes"] = {
+                "Institution-Specific Code": institute_code_list[i]
+            }
+
+        return mock_data
+
+    @mock.patch(
+        "process_report.processors.coldfront_fetch_processor.ColdfrontFetchProcessor._fetch_coldfront_allocation_api",
+    )
+    def test_coldfront_fetch(self, mock_get_allocation_data):
+        mock_get_allocation_data.return_value = self._get_mock_allocation_data(
+            ["P1", "P2", "P3", "P4"],
+            ["PI1", "PI1", "", "PI12"],
+            ["IC1", "", "", "IC2"],
+        )
+        test_invoice = self._get_test_invoice(["P1", "P1", "P2", "P3", "P4"])
+        answer_invoice = self._get_test_invoice(
+            ["P1", "P1", "P2", "P3", "P4"],
+            ["PI1", "PI1", "PI1", "", "PI12"],
+            ["IC1", "IC1", "", "", "IC2"],
+        )
+        test_coldfront_fetch_proc = test_utils.new_coldfront_fetch_processor(
+            data=test_invoice
+        )
+        test_coldfront_fetch_proc.process()
+        output_invoice = test_coldfront_fetch_proc.data
+        self.assertTrue(output_invoice.equals(answer_invoice))
+
+    @mock.patch(
+        "process_report.processors.coldfront_fetch_processor.ColdfrontFetchProcessor._fetch_coldfront_allocation_api",
+    )
+    def test_coldfront_project_not_found(self, mock_get_allocation_data):
+        """What happens when an invoice project is not found in Coldfront."""
+        mock_get_allocation_data.return_value = self._get_mock_allocation_data(
+            ["P1", "P2"],
+            ["PI1", "PI1"],
+            ["IC1", "IC2"],
+        )
+        test_nonbillable_projects = ["P3"]
+        test_invoice = self._get_test_invoice(["P1", "P2", "P3", "P4", "P5"])
+        answer_project_set = ["P4", "P5"]
+        test_coldfront_fetch_proc = test_utils.new_coldfront_fetch_processor(
+            data=test_invoice, nonbillable_projects=test_nonbillable_projects
+        )
+
+        with self.assertLogs() as log:
+            test_coldfront_fetch_proc.process()
+            self.assertIn(
+                f"Projects {answer_project_set} not found in Coldfront and are billable! Please check the project names",
+                log.output[0],
+            )
+
+    @mock.patch("requests.sessions.Session.get")
+    def test_query_url(self, mock_request_get):
+        test_coldfront_fetch_proc = test_utils.new_coldfront_fetch_processor()
+        test_coldfront_fetch_proc.coldfront_client = mock.MagicMock()
+        test_coldfront_fetch_proc._fetch_coldfront_allocation_api(["P1", "P2"])
+        test_coldfront_fetch_proc.coldfront_client.get.assert_called_with(
+            "https://coldfront.mss.mghpcc.org/api/allocations?attr_Allocated Project Name=P1&attr_Allocated Project Name=P2"
+        )
diff --git a/process_report/tests/unit/test_util.py b/process_report/tests/unit/test_util.py
index f75e6819..e5496a0e 100644
--- a/process_report/tests/unit/test_util.py
+++ b/process_report/tests/unit/test_util.py
@@ -1,4 +1,4 @@
-from unittest import TestCase
+from unittest import TestCase, mock
 import tempfile
 import pandas
 import os
@@ -104,3 +104,16 @@ def test_timed_projects(self):
 
         expected_projects = ["ProjectB", "ProjectC", "ProjectD"]
         self.assertEqual(excluded_projects, expected_projects)
+
+
+class TestValidateRequiredEnvVars(TestCase):
+    @mock.patch.dict(
+        "os.environ", {"KEYCLOAK_CLIENT_ID": "test", "KEYCLOAK_CLIENT_SECRET": "test"}
+    )
+    def test_env_vars_valid(self):
+        process_report.validate_required_env_vars()
+
+    @mock.patch.dict("os.environ", {"KEYCLOAK_CLIENT_ID": "test"})
+    def test_env_vars_missing(self):
+        with self.assertRaises(SystemExit):
+            process_report.validate_required_env_vars()
diff --git a/process_report/tests/util.py b/process_report/tests/util.py
index abbe5dbf..e18ab30a 100644
--- a/process_report/tests/util.py
+++ b/process_report/tests/util.py
@@ -8,6 +8,7 @@
 )
 
 from process_report.processors import (
+    coldfront_fetch_processor,
     add_institution_processor,
     validate_pi_alias_processor,
     lenovo_processor,
@@ -66,6 +67,21 @@ def new_pi_specific_invoice(
     )
 
 
+def new_coldfront_fetch_processor(
+    name="",
+    invoice_month="0000-00",
+    data=None,
+    nonbillable_projects=None,
+):
+    if data is None:
+        data = pandas.DataFrame()
+    if nonbillable_projects is None:
+        nonbillable_projects = []
+    return coldfront_fetch_processor.ColdfrontFetchProcessor(
+        name, invoice_month, data, nonbillable_projects
+    )
+
+
 def new_add_institution_processor(
     name="",
     invoice_month="0000-00",


Reviews:
user: login: naved001
state: COMMENTED
author_association: COLLABORATOR
submitted_at: 2025-03-07T18:25:20Z
user: login: knikolla
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-03-10T22:31:48Z
user: login: larsks
state: COMMENTED
author_association: MEMBER
submitted_at: 2025-03-11T17:49:31Z
user: login: QuanMPhm
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-03-11T19:09:20Z
user: login: naved001
state: COMMENTED
author_association: COLLABORATOR
submitted_at: 2025-03-18T17:55:32Z
user: login: QuanMPhm
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-03-20T18:04:54Z
user: login: larsks
state: APPROVED
author_association: MEMBER
submitted_at: 2025-03-20T18:24:35Z
user: login: QuanMPhm
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-03-21T02:22:54Z
user: login: larsks
state: COMMENTED
author_association: MEMBER
submitted_at: 2025-03-21T04:14:16Z
user: login: larsks
body: There is a problem with one of the tests. Occasionally, `process_report/tests/unit/processors/test_coldfront_fetch_processor.py::TestColdfrontFetchProcessor::test_coldfront_project_not_found` fails with the following error:

> FAILED process_report/tests/unit/processors/test_coldfront_fetch_processor.py::TestColdfrontFetchProcessor::test_coldfront_project_not_found - AssertionError: "Projects {'P5', 'P4'} not found in Coldfront and are billable! Please check the project names" not found in "WARNING:process_report.processors.coldfront_fetch_processor:Projects {'P4', 'P5'} not found in Coldfront and are billable! Please check the project names"

~It's too late at night for me to figure out what's going on here, but~ when I ran this test 100 times just now it failed 3 times:

```
$ ok=0; failed=0; for _ in {1..100}; do if uv run pytest process_report/tests/unit/processors/test_coldfront_fetch_processor.py; then let ok++; else let failed++; fi ; done; echo ok $ok failed $failed
ok 97 failed 3
```

On a second run, it worked 99 times and failed 1 time.

We should figure out what's going on here. This is either a buggy test, which will lead to erroneous CI failures, or it's an actual bug in the code.

---

Update: it's an ordering error. When the test fails, `log.output[0]` contains:

```
(Pdb) log.output[0]
"WARNING:process_report.processors.coldfront_fetch_processor:Projects {'P4', 'P5'} not found in Coldfront and are billable! Please check the project names"
```

But we're checking it against:

```
(Pdb) f"Projects {answer_project_set} not found in Coldfront and are billable! Please check the project names"
"Projects {'P5', 'P4'} not found in Coldfront and are billable! Please check the project names"
```

You need to ensure that the list of projects in the error message is always presented in the same order.
state: CHANGES_REQUESTED
author_association: MEMBER
submitted_at: 2025-03-21T04:27:38Z
user: login: naved001
state: COMMENTED
author_association: COLLABORATOR
submitted_at: 2025-04-04T15:31:42Z
user: login: larsks
state: APPROVED
author_association: MEMBER
submitted_at: 2025-04-04T17:21:37Z
user: login: naved001
state: APPROVED
author_association: COLLABORATOR
submitted_at: 2025-04-08T16:27:20Z
user: login: QuanMPhm
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-04-08T18:09:31Z

Review comments:
diff_hunk: @@ -0,0 +1,104 @@
+import os
+import functools
+import logging
+from dataclasses import dataclass
+
+import requests
+
+from process_report.invoices import invoice
+from process_report.processors import processor
+
+logger = logging.getLogger(__name__)
+logging.basicConfig(level=logging.INFO)
+
+
+CF_ATTR_ALLOCATED_PROJECT_NAME = "Allocated Project Name"
+
+
+@dataclass
+class ColdfrontFetchProcessor(processor.Processor):
+    nonbillable_projects: list[str]
+
+    @functools.cached_property
+    def coldfront_client(self):
+        keycloak_url = os.environ.get("KEYCLOAK_URL", "https://keycloak.mss.mghpcc.org")
+
+        # Authenticate with Keycloak
+        token_url = f"{keycloak_url}/auth/realms/mss/protocol/openid-connect/token"
+        try:
+            r = requests.post(
+                token_url,
+                data={"grant_type": "client_credentials"},
+                auth=requests.auth.HTTPBasicAuth(
+                    os.environ["KEYCLOAK_CLIENT_ID"],
+                    os.environ["KEYCLOAK_CLIENT_SECRET"],
+                ),
+            )
+        except KeyError:
+            logger.error(
+                "Keycloak authentication failed. Please set the environment variables KEYCLOAK_CLIENT_ID and KEYCLOAK_CLIENT_ID"
+            )
+
+        client_token = r.json()["access_token"]
+
+        session = requests.session()
+        headers = {
+            "Authorization": f"Bearer {client_token}",
+            "Content-Type": "application/json",
+        }
+        session.headers.update(headers)
+        return session
+
+    def _get_project_list(self):
+        return self.data[invoice.PROJECT_FIELD].unique()
+
+    def _fetch_coldfront_allocation_api(self, allocation_list):
+        coldfront_api_url = os.environ.get(
+            "COLDFRONT_URL", "https://coldfront.mss.mghpcc.org/api/allocations"
+        )
+        api_query_str = "&".join(
+            [
+                f"attr_{CF_ATTR_ALLOCATED_PROJECT_NAME}={project}"
+                for project in allocation_list
+            ]
+        )
+        r = self.coldfront_client.get(f"{coldfront_api_url}?{api_query_str}")
path: process_report/processors/coldfront_fetch_processor.py
user: login: naved001
body: so it looks like instead of asking for all allocations, we are going to pass in a giant list of all the allocations in our report. Is this really that much more scalable? 
created_at: 2025-03-07T18:22:19Z
author_association: COLLABORATOR
diff_hunk: @@ -0,0 +1,104 @@
+import os
+import functools
+import logging
+from dataclasses import dataclass
+
+import requests
+
+from process_report.invoices import invoice
+from process_report.processors import processor
+
+logger = logging.getLogger(__name__)
+logging.basicConfig(level=logging.INFO)
+
+
+CF_ATTR_ALLOCATED_PROJECT_NAME = "Allocated Project Name"
+
+
+@dataclass
+class ColdfrontFetchProcessor(processor.Processor):
+    nonbillable_projects: list[str]
+
+    @functools.cached_property
+    def coldfront_client(self):
+        keycloak_url = os.environ.get("KEYCLOAK_URL", "https://keycloak.mss.mghpcc.org")
+
+        # Authenticate with Keycloak
+        token_url = f"{keycloak_url}/auth/realms/mss/protocol/openid-connect/token"
+        try:
+            r = requests.post(
+                token_url,
+                data={"grant_type": "client_credentials"},
+                auth=requests.auth.HTTPBasicAuth(
+                    os.environ["KEYCLOAK_CLIENT_ID"],
+                    os.environ["KEYCLOAK_CLIENT_SECRET"],
+                ),
+            )
+        except KeyError:
+            logger.error(
+                "Keycloak authentication failed. Please set the environment variables KEYCLOAK_CLIENT_ID and KEYCLOAK_CLIENT_ID"
+            )
+
+        client_token = r.json()["access_token"]
+
+        session = requests.session()
+        headers = {
+            "Authorization": f"Bearer {client_token}",
+            "Content-Type": "application/json",
+        }
+        session.headers.update(headers)
+        return session
+
+    def _get_project_list(self):
+        return self.data[invoice.PROJECT_FIELD].unique()
+
+    def _fetch_coldfront_allocation_api(self, allocation_list):
+        coldfront_api_url = os.environ.get(
+            "COLDFRONT_URL", "https://coldfront.mss.mghpcc.org/api/allocations"
+        )
+        api_query_str = "&".join(
+            [
+                f"attr_{CF_ATTR_ALLOCATED_PROJECT_NAME}={project}"
+                for project in allocation_list
+            ]
+        )
+        r = self.coldfront_client.get(f"{coldfront_api_url}?{api_query_str}")
path: process_report/processors/coldfront_fetch_processor.py
user: login: knikolla
body: @naved001 That's a valid concern. I believe it is pretty scalable up to a certain point.

Each allocation will require 28 characters for `attr_Allocated Project Name=` + 64 characters for max allocation name length <= 100 characters total per allocation query.

Assuming 100 allocations active * 100 per allocation query = 10 000.

At 1byte per char that's 10kb, which is already much less in size than the complete list of all allocations.

My only concern would be the whether nginx ingress controller is configured or supports that size of URIs out of the box.

@QuanMPhm can you do some edge testing in your openshift cluster with some really high number of allocations? say 100 and then 1000?
created_at: 2025-03-10T22:31:48Z
author_association: CONTRIBUTOR
diff_hunk: @@ -0,0 +1,104 @@
+import os
+import functools
+import logging
+from dataclasses import dataclass
+
+import requests
+
+from process_report.invoices import invoice
+from process_report.processors import processor
+
+logger = logging.getLogger(__name__)
+logging.basicConfig(level=logging.INFO)
+
+
+CF_ATTR_ALLOCATED_PROJECT_NAME = "Allocated Project Name"
+
+
+@dataclass
+class ColdfrontFetchProcessor(processor.Processor):
+    nonbillable_projects: list[str]
+
+    @functools.cached_property
+    def coldfront_client(self):
+        keycloak_url = os.environ.get("KEYCLOAK_URL", "https://keycloak.mss.mghpcc.org")
+
+        # Authenticate with Keycloak
+        token_url = f"{keycloak_url}/auth/realms/mss/protocol/openid-connect/token"
+        try:
+            r = requests.post(
+                token_url,
+                data={"grant_type": "client_credentials"},
+                auth=requests.auth.HTTPBasicAuth(
+                    os.environ["KEYCLOAK_CLIENT_ID"],
+                    os.environ["KEYCLOAK_CLIENT_SECRET"],
+                ),
+            )
+        except KeyError:
+            logger.error(
+                "Keycloak authentication failed. Please set the environment variables KEYCLOAK_CLIENT_ID and KEYCLOAK_CLIENT_ID"
+            )
path: process_report/processors/coldfront_fetch_processor.py
user: login: larsks
body: You log an error here, but then just keep moving forward. Is that intentional, or should you abort the operation here?
created_at: 2025-03-11T17:45:13Z
author_association: MEMBER
diff_hunk: @@ -0,0 +1,104 @@
+import os
+import functools
+import logging
+from dataclasses import dataclass
+
+import requests
+
+from process_report.invoices import invoice
+from process_report.processors import processor
+
+logger = logging.getLogger(__name__)
+logging.basicConfig(level=logging.INFO)
+
+
+CF_ATTR_ALLOCATED_PROJECT_NAME = "Allocated Project Name"
+
+
+@dataclass
+class ColdfrontFetchProcessor(processor.Processor):
+    nonbillable_projects: list[str]
+
+    @functools.cached_property
+    def coldfront_client(self):
+        keycloak_url = os.environ.get("KEYCLOAK_URL", "https://keycloak.mss.mghpcc.org")
+
+        # Authenticate with Keycloak
+        token_url = f"{keycloak_url}/auth/realms/mss/protocol/openid-connect/token"
+        try:
+            r = requests.post(
+                token_url,
+                data={"grant_type": "client_credentials"},
+                auth=requests.auth.HTTPBasicAuth(
+                    os.environ["KEYCLOAK_CLIENT_ID"],
+                    os.environ["KEYCLOAK_CLIENT_SECRET"],
+                ),
+            )
path: process_report/processors/coldfront_fetch_processor.py
user: login: larsks
body: You never check for HTTP errors. One option is to call `r.raise_for_status()` inside the `try/except` block, and then add an `except` stanza that will translate http errors into a more user friendly error message.
created_at: 2025-03-11T17:45:58Z
author_association: MEMBER
diff_hunk: @@ -0,0 +1,104 @@
+import os
+import functools
+import logging
+from dataclasses import dataclass
+
+import requests
+
+from process_report.invoices import invoice
+from process_report.processors import processor
+
+logger = logging.getLogger(__name__)
+logging.basicConfig(level=logging.INFO)
+
+
+CF_ATTR_ALLOCATED_PROJECT_NAME = "Allocated Project Name"
+
+
+@dataclass
+class ColdfrontFetchProcessor(processor.Processor):
+    nonbillable_projects: list[str]
+
+    @functools.cached_property
+    def coldfront_client(self):
+        keycloak_url = os.environ.get("KEYCLOAK_URL", "https://keycloak.mss.mghpcc.org")
+
+        # Authenticate with Keycloak
+        token_url = f"{keycloak_url}/auth/realms/mss/protocol/openid-connect/token"
+        try:
+            r = requests.post(
+                token_url,
+                data={"grant_type": "client_credentials"},
+                auth=requests.auth.HTTPBasicAuth(
+                    os.environ["KEYCLOAK_CLIENT_ID"],
+                    os.environ["KEYCLOAK_CLIENT_SECRET"],
+                ),
+            )
+        except KeyError:
+            logger.error(
+                "Keycloak authentication failed. Please set the environment variables KEYCLOAK_CLIENT_ID and KEYCLOAK_CLIENT_ID"
+            )
path: process_report/processors/coldfront_fetch_processor.py
user: login: larsks
body: If these environment variables are *required*, should we be checking for them at an earlier point in program execution?
created_at: 2025-03-11T17:46:46Z
author_association: MEMBER
diff_hunk: @@ -0,0 +1,104 @@
+import os
+import functools
+import logging
+from dataclasses import dataclass
+
+import requests
+
+from process_report.invoices import invoice
+from process_report.processors import processor
+
+logger = logging.getLogger(__name__)
+logging.basicConfig(level=logging.INFO)
+
+
+CF_ATTR_ALLOCATED_PROJECT_NAME = "Allocated Project Name"
+
+
+@dataclass
+class ColdfrontFetchProcessor(processor.Processor):
+    nonbillable_projects: list[str]
+
+    @functools.cached_property
+    def coldfront_client(self):
+        keycloak_url = os.environ.get("KEYCLOAK_URL", "https://keycloak.mss.mghpcc.org")
+
+        # Authenticate with Keycloak
+        token_url = f"{keycloak_url}/auth/realms/mss/protocol/openid-connect/token"
+        try:
+            r = requests.post(
+                token_url,
+                data={"grant_type": "client_credentials"},
+                auth=requests.auth.HTTPBasicAuth(
+                    os.environ["KEYCLOAK_CLIENT_ID"],
+                    os.environ["KEYCLOAK_CLIENT_SECRET"],
+                ),
+            )
+        except KeyError:
+            logger.error(
+                "Keycloak authentication failed. Please set the environment variables KEYCLOAK_CLIENT_ID and KEYCLOAK_CLIENT_ID"
+            )
+
+        client_token = r.json()["access_token"]
+
+        session = requests.session()
+        headers = {
+            "Authorization": f"Bearer {client_token}",
+            "Content-Type": "application/json",
+        }
+        session.headers.update(headers)
+        return session
+
+    def _get_project_list(self):
+        return self.data[invoice.PROJECT_FIELD].unique()
+
+    def _fetch_coldfront_allocation_api(self, allocation_list):
+        coldfront_api_url = os.environ.get(
+            "COLDFRONT_URL", "https://coldfront.mss.mghpcc.org/api/allocations"
+        )
+        api_query_str = "&".join(
+            [
+                f"attr_{CF_ATTR_ALLOCATED_PROJECT_NAME}={project}"
+                for project in allocation_list
+            ]
+        )
+        r = self.coldfront_client.get(f"{coldfront_api_url}?{api_query_str}")
path: process_report/processors/coldfront_fetch_processor.py
user: login: QuanMPhm
body: @knikolla I wrote a script that queried a coldfront deployment running in my local Openshift cluster. When quyerying 100 allocations (i.e `/allocations?allo=1&allo=2&...`), the query took 0.3s. At 1000 allocations, the query took 2.3s. Any more allocations beyond that, and we hit [Django's default query field limit](https://docs.djangoproject.com/en/5.1/ref/settings/#data-upload-max-number-fields).

@larsks Mentioned that this maybe something we should look into the future. Perhaps we could try sending the allocation list in the request payload instead of encoding it in the URL.
created_at: 2025-03-11T19:09:20Z
author_association: CONTRIBUTOR
diff_hunk: @@ -0,0 +1,104 @@
+import os
+import functools
+import logging
+from dataclasses import dataclass
+
+import requests
+
+from process_report.invoices import invoice
+from process_report.processors import processor
+
+logger = logging.getLogger(__name__)
+logging.basicConfig(level=logging.INFO)
+
+
+CF_ATTR_ALLOCATED_PROJECT_NAME = "Allocated Project Name"
+
+
+@dataclass
+class ColdfrontFetchProcessor(processor.Processor):
+    nonbillable_projects: list[str]
+
+    @functools.cached_property
+    def coldfront_client(self):
+        keycloak_url = os.environ.get("KEYCLOAK_URL", "https://keycloak.mss.mghpcc.org")
+
+        # Authenticate with Keycloak
+        token_url = f"{keycloak_url}/auth/realms/mss/protocol/openid-connect/token"
+        try:
+            r = requests.post(
+                token_url,
+                data={"grant_type": "client_credentials"},
+                auth=requests.auth.HTTPBasicAuth(
+                    os.environ["KEYCLOAK_CLIENT_ID"],
+                    os.environ["KEYCLOAK_CLIENT_SECRET"],
+                ),
+            )
path: process_report/processors/coldfront_fetch_processor.py
user: login: naved001
body: @QuanMPhm ^^ Please check for errors and log a useful error message.

Somebody accidentally changed my secret (which sets the KEYCLOAK_CLIENT* env variables) and I was wondering why my openshift report was failing. If it logged the http message (in my case 401 for unauthorized) and the client_id I would have immediately spotted the problem instead of debugging the code in the container because it worked fine on my local machine.


created_at: 2025-03-18T17:55:32Z
author_association: COLLABORATOR
diff_hunk: @@ -0,0 +1,104 @@
+import os
+import functools
+import logging
+from dataclasses import dataclass
+
+import requests
+
+from process_report.invoices import invoice
+from process_report.processors import processor
+
+logger = logging.getLogger(__name__)
+logging.basicConfig(level=logging.INFO)
+
+
+CF_ATTR_ALLOCATED_PROJECT_NAME = "Allocated Project Name"
+
+
+@dataclass
+class ColdfrontFetchProcessor(processor.Processor):
+    nonbillable_projects: list[str]
+
+    @functools.cached_property
+    def coldfront_client(self):
+        keycloak_url = os.environ.get("KEYCLOAK_URL", "https://keycloak.mss.mghpcc.org")
+
+        # Authenticate with Keycloak
+        token_url = f"{keycloak_url}/auth/realms/mss/protocol/openid-connect/token"
+        try:
+            r = requests.post(
+                token_url,
+                data={"grant_type": "client_credentials"},
+                auth=requests.auth.HTTPBasicAuth(
+                    os.environ["KEYCLOAK_CLIENT_ID"],
+                    os.environ["KEYCLOAK_CLIENT_SECRET"],
+                ),
+            )
+        except KeyError:
+            logger.error(
+                "Keycloak authentication failed. Please set the environment variables KEYCLOAK_CLIENT_ID and KEYCLOAK_CLIENT_ID"
+            )
path: process_report/processors/coldfront_fetch_processor.py
user: login: QuanMPhm
body: I will add a function in `process_report.process_report.py` that will check for all required env vars. I will have this function called at the start of `main()` before the s3 input files are fetched.
created_at: 2025-03-20T18:04:54Z
author_association: CONTRIBUTOR
diff_hunk: @@ -95,6 +99,12 @@ def get_iso8601_time():
     return datetime.datetime.now().strftime("%Y%m%dT%H%M%SZ")
 
 
+def validate_required_env_vars():
path: process_report/process_report.py
user: login: larsks
body: This function appears to be untested.
created_at: 2025-03-20T18:21:32Z
author_association: MEMBER
diff_hunk: @@ -95,6 +99,12 @@ def get_iso8601_time():
     return datetime.datetime.now().strftime("%Y%m%dT%H%M%SZ")
 
 
+def validate_required_env_vars():
path: process_report/process_report.py
user: login: QuanMPhm
body: I added a unit test in `test_util.py`, though given your approval, should I understand that this suggestion was optional?

Also, if you don't mind me rambling on, when does a piece of code become complex enough to warrant a test? I thought this function was too simple to warrant a test. Also, within this repo and a few other repos I worked on, not every function has a test case. 
created_at: 2025-03-21T02:22:54Z
author_association: CONTRIBUTOR
diff_hunk: @@ -95,6 +99,12 @@ def get_iso8601_time():
     return datetime.datetime.now().strftime("%Y%m%dT%H%M%SZ")
 
 
+def validate_required_env_vars():
path: process_report/process_report.py
user: login: larsks
body: In an interpreted, dynamically typed language like Python, if your code doesn't execute during your tests then it could be hiding errors that will crop up at runtime. Consider what would happen if `validate_required_env_vars` looked like this:

```
def validate_required_env_vars():
    for required_env_var in REQUIRED_ENV_VAR:
        if required_env_var not in os.environ:
            sys.exit(f"Required environment variable {required_env_var} is not set")
```

There's a typo there -- `REQUIRED_ENV_VARS` is mis-spelled. As it stands, all your tests pass despite this error. That allows it to sit there undetected until somebody actually runs the code.

The goal of unit tests is to prevent this situation.

You don't necessarily need a test for every individual function, but you do need to ensure that as much code is possible is run as part of your tests. This is where tools like `coverage` come in very handy; if we run the tests using `pytest` and the `coverage` plugin...

```
pytest --cov=process_report --cov-report=html
```

...then we get a nice HTML report that shows exactly what lines of code were covered by tests and what lines were not.

created_at: 2025-03-21T04:14:16Z
author_association: MEMBER
diff_hunk: @@ -0,0 +1,104 @@
+import os
+import functools
+import logging
+from dataclasses import dataclass
+
+import requests
+
+from process_report.invoices import invoice
+from process_report.processors import processor
+
+logger = logging.getLogger(__name__)
+logging.basicConfig(level=logging.INFO)
+
+
+CF_ATTR_ALLOCATED_PROJECT_NAME = "Allocated Project Name"
+
+
+@dataclass
+class ColdfrontFetchProcessor(processor.Processor):
+    nonbillable_projects: list[str]
+
+    @functools.cached_property
+    def coldfront_client(self):
+        keycloak_url = os.environ.get("KEYCLOAK_URL", "https://keycloak.mss.mghpcc.org")
+
+        # Authenticate with Keycloak
+        token_url = f"{keycloak_url}/auth/realms/mss/protocol/openid-connect/token"
+        try:
+            r = requests.post(
+                token_url,
+                data={"grant_type": "client_credentials"},
+                auth=requests.auth.HTTPBasicAuth(
+                    os.environ["KEYCLOAK_CLIENT_ID"],
+                    os.environ["KEYCLOAK_CLIENT_SECRET"],
+                ),
+            )
+        except KeyError:
+            logger.error(
+                "Keycloak authentication failed. Please set the environment variables KEYCLOAK_CLIENT_ID and KEYCLOAK_CLIENT_ID"
+            )
+
+        client_token = r.json()["access_token"]
+
+        session = requests.session()
+        headers = {
+            "Authorization": f"Bearer {client_token}",
+            "Content-Type": "application/json",
+        }
+        session.headers.update(headers)
+        return session
+
+    def _get_project_list(self):
+        return self.data[invoice.PROJECT_FIELD].unique()
+
+    def _fetch_coldfront_allocation_api(self, allocation_list):
+        coldfront_api_url = os.environ.get(
+            "COLDFRONT_URL", "https://coldfront.mss.mghpcc.org/api/allocations"
+        )
+        api_query_str = "&".join(
+            [
+                f"attr_{CF_ATTR_ALLOCATED_PROJECT_NAME}={project}"
+                for project in allocation_list
+            ]
+        )
+        r = self.coldfront_client.get(f"{coldfront_api_url}?{api_query_str}")
path: process_report/processors/coldfront_fetch_processor.py
user: login: naved001
body: > Mentioned that this maybe something we should look into the future. Perhaps we could try sending the allocation list in the request payload instead of encoding it in the URL.

This seems reasonable to me.
created_at: 2025-04-02T19:17:58Z
author_association: COLLABORATOR
diff_hunk: @@ -0,0 +1,106 @@
+import os
+import sys
+import functools
+import logging
+from dataclasses import dataclass
+
+import requests
+
+from process_report.invoices import invoice
+from process_report.processors import processor
+
+logger = logging.getLogger(__name__)
+logging.basicConfig(level=logging.INFO)
+
+
+CF_ATTR_ALLOCATED_PROJECT_NAME = "Allocated Project Name"
+
+
+@dataclass
+class ColdfrontFetchProcessor(processor.Processor):
+    nonbillable_projects: list[str]
+
+    @functools.cached_property
+    def coldfront_client(self):
+        keycloak_url = os.environ.get("KEYCLOAK_URL", "https://keycloak.mss.mghpcc.org")
path: process_report/processors/coldfront_fetch_processor.py
user: login: larsks
body: I'm not suggesting that you change this now, but in general a good design principle is get configuration from the environment *outside* of the class (e.g., in your `main` method), and then pass that information into the class as class parameters. This makes things easier to test (you can just instantiate classes with your choice of values), and leads to better reuse (what if want to get values from a configuration file rather than the environment?).
created_at: 2025-04-04T17:19:49Z
author_association: MEMBER
diff_hunk: @@ -0,0 +1,92 @@
+from unittest import TestCase, mock
+import pandas
+
+from process_report.tests import util as test_utils
+
+
+class TestColdfrontFetchProcessor(TestCase):
+    def _get_test_invoice(
+        self,
+        project,
+        pi=None,
+        institute_code=None,
+    ):
+        if not pi:
+            pi = [""] * len(project)
+
+        if not institute_code:
+            institute_code = [""] * len(project)
+
+        return pandas.DataFrame(
+            {
+                "Manager (PI)": pi,
+                "Project - Allocation": project,
+                "Institution - Specific Code": institute_code,
+            }
+        )
+
+    def _get_mock_allocation_data(self, project_list, pi_list, institute_code_list):
+        mock_data = {}
+        for i, project in enumerate(project_list):
+            mock_data[project] = {}
+            mock_data[project]["project"] = {"pi": pi_list[i]}
+            mock_data[project]["attributes"] = {
+                "Institution-Specific Code": institute_code_list[i]
+            }
+
+        return mock_data
+
+    @mock.patch(
+        "process_report.processors.coldfront_fetch_processor.ColdfrontFetchProcessor._fetch_coldfront_allocation_api",
+    )
+    def test_coldfront_fetch(self, mock_get_allocation_data):
+        mock_get_allocation_data.return_value = self._get_mock_allocation_data(
+            ["P1", "P2", "P3", "P4"],
+            ["PI1", "PI1", "", "PI12"],
+            ["IC1", "", "", "IC2"],
path: process_report/tests/unit/processors/test_coldfront_fetch_processor.py
user: login: naved001
body: Just so I understand: you are associating `PI1` with Project `P2` but then it doesn't have an Institution Code. Is the institution code associated with the Project or the PI? Probably doesn't matter for the test, but I would like it to match the real world data.
created_at: 2025-04-08T16:19:12Z
author_association: COLLABORATOR
diff_hunk: @@ -0,0 +1,92 @@
+from unittest import TestCase, mock
+import pandas
+
+from process_report.tests import util as test_utils
+
+
+class TestColdfrontFetchProcessor(TestCase):
+    def _get_test_invoice(
+        self,
+        project,
+        pi=None,
+        institute_code=None,
+    ):
+        if not pi:
+            pi = [""] * len(project)
+
+        if not institute_code:
+            institute_code = [""] * len(project)
+
+        return pandas.DataFrame(
+            {
+                "Manager (PI)": pi,
+                "Project - Allocation": project,
+                "Institution - Specific Code": institute_code,
+            }
+        )
+
+    def _get_mock_allocation_data(self, project_list, pi_list, institute_code_list):
+        mock_data = {}
+        for i, project in enumerate(project_list):
+            mock_data[project] = {}
+            mock_data[project]["project"] = {"pi": pi_list[i]}
+            mock_data[project]["attributes"] = {
+                "Institution-Specific Code": institute_code_list[i]
+            }
+
+        return mock_data
+
+    @mock.patch(
+        "process_report.processors.coldfront_fetch_processor.ColdfrontFetchProcessor._fetch_coldfront_allocation_api",
+    )
+    def test_coldfront_fetch(self, mock_get_allocation_data):
+        mock_get_allocation_data.return_value = self._get_mock_allocation_data(
+            ["P1", "P2", "P3", "P4"],
+            ["PI1", "PI1", "", "PI12"],
+            ["IC1", "", "", "IC2"],
path: process_report/tests/unit/processors/test_coldfront_fetch_processor.py
user: login: QuanMPhm
body: From the invoicing requirements file, the institute code is unique to each project, so I guess this test data could be real world data.
created_at: 2025-04-08T18:09:31Z
author_association: CONTRIBUTOR

PR comments:
user: login: QuanMPhm
created_at: 2025-03-21T14:05:04Z
body: Thank you Mr.@larsks for taking the time to investigate this bug. I learned something about Python sets today. I've converted the set of missing projects into an ordered list.
author_association: CONTRIBUTOR
user: login: QuanMPhm
created_at: 2025-04-08T15:27:23Z
body: @knikolla @naved001 If it's fine, I would like your input before merging
author_association: CONTRIBUTOR
user: login: QuanMPhm
created_at: 2025-04-10T14:21:34Z
body: This is dependant on both https://github.com/CCI-MOC/openshift-usage-scripts/issues/111 and https://github.com/CCI-MOC/openstack-billing-from-db/issues/91
author_association: CONTRIBUTOR
user: login: knikolla
created_at: 2025-04-29T21:24:42Z
body: > This is dependant on both [CCI-MOC/openshift-usage-scripts#111](https://github.com/CCI-MOC/openshift-usage-scripts/issues/111) and [CCI-MOC/openstack-billing-from-db#91](https://github.com/CCI-MOC/openstack-billing-from-db/issues/91)

@QuanMPhm It's the opposite, those are dependent on this. If this has required approval, merge. 
author_association: CONTRIBUTOR