Repository: unicef/hope
Pull request #4708: Dashboard Hotfix
Author: login: srugano
Labels: name: hotfix
name: backend
description: Changes in BE code
Created: 2025-03-21T20:50:39Z | Merged: 2025-03-26T08:35:24Z

[AB#241723](https://unicef.visualstudio.com/ICTD-HCT-MIS/_workitems/edit/241723)


Diff:
diff --git a/pyproject.toml b/pyproject.toml
index ad18e03dbe2..f568e2fdeff 100644
--- a/pyproject.toml
+++ b/pyproject.toml
@@ -132,7 +132,7 @@ distribution = true
 
 [project]
 name = "hope"
-version = "3.1.5"
+version = "3.1.6"
 description = "HCT MIS is UNICEF's humanitarian cash transfer platform."
 authors = [
     { name = "Tivix" },
diff --git a/src/frontend/package.json b/src/frontend/package.json
index 50b675ed79f..844b08a4c7c 100644
--- a/src/frontend/package.json
+++ b/src/frontend/package.json
@@ -1,6 +1,6 @@
 {
   "name": "frontend",
-  "version": "3.1.5",
+  "version": "3.1.6",
   "private": true,
   "type": "module",
   "scripts": {
diff --git a/src/hct_mis_api/apps/dashboard/services.py b/src/hct_mis_api/apps/dashboard/services.py
index 7e54bc34b0f..3a6f4b866f4 100644
--- a/src/hct_mis_api/apps/dashboard/services.py
+++ b/src/hct_mis_api/apps/dashboard/services.py
@@ -91,6 +91,8 @@ def refresh_data(cls, business_area_slug: str) -> ReturnDict:
                 .filter(
                     business_area=area,
                     parent__status__in=["ACCEPTED", "FINISHED"],
+                    program__is_visible=True,
+                    parent__is_removed=False,
                     is_removed=False,
                     conflicted=False,
                 )  # noqa
@@ -122,7 +124,11 @@ def refresh_data(cls, business_area_slug: str) -> ReturnDict:
                     total_usd=Sum(
                         Case(
                             When(delivered_quantity_usd__isnull=False, then="delivered_quantity_usd"),
-                            When(entitlement_quantity_usd__isnull=False, then="entitlement_quantity_usd"),
+                            When(
+                                delivered_quantity_usd__isnull=True,
+                                entitlement_quantity_usd__isnull=False,
+                                then="entitlement_quantity_usd",
+                            ),
                             default=Value(0.0),
                             output_field=DecimalField(),
                         )
@@ -130,7 +136,11 @@ def refresh_data(cls, business_area_slug: str) -> ReturnDict:
                     total_quantity=Sum(
                         Case(
                             When(delivered_quantity__isnull=False, then="delivered_quantity"),
-                            When(entitlement_quantity__isnull=False, then="entitlement_quantity"),
+                            When(
+                                delivered_quantity__isnull=True,
+                                entitlement_quantity__isnull=False,
+                                then="entitlement_quantity",
+                            ),
                             default=Value(0.0),
                             output_field=DecimalField(),
                         )
@@ -234,6 +244,8 @@ def refresh_data(cls, business_area_slug: str = "global") -> ReturnDict:
             )
             .filter(
                 parent__status__in=["ACCEPTED", "FINISHED"],
+                program__is_visible=True,
+                parent__is_removed=False,
                 is_removed=False,
                 conflicted=False,
             )  # noqa
@@ -257,7 +269,11 @@ def refresh_data(cls, business_area_slug: str = "global") -> ReturnDict:
                 total_usd=Sum(
                     Case(
                         When(delivered_quantity_usd__isnull=False, then="delivered_quantity_usd"),
-                        When(entitlement_quantity_usd__isnull=False, then="entitlement_quantity_usd"),
+                        When(
+                            delivered_quantity_usd__isnull=True,
+                            entitlement_quantity_usd__isnull=False,
+                            then="entitlement_quantity_usd",
+                        ),
                         default=Value(0.0),
                         output_field=DecimalField(),
                     )
diff --git a/tests/selenium/conftest.py b/tests/selenium/conftest.py
index db484bfdf6f..fcb3b47d41d 100644
--- a/tests/selenium/conftest.py
+++ b/tests/selenium/conftest.py
@@ -313,6 +313,9 @@ def login(browser: Chrome) -> Chrome:
     browser.find_element(By.ID, login).send_keys("superuser")
     browser.find_element(By.ID, password).send_keys("testtest2")
     browser.find_element(By.XPATH, loginButton).click()
+    from time import sleep
+
+    sleep(0.2)
     browser.get(f"{browser.live_server.url}/")
     from django.core.cache import cache
 
diff --git a/tests/unit/apps/dashboard/test_services.py b/tests/unit/apps/dashboard/test_services.py
index f14fe89d104..a5187cdc802 100644
--- a/tests/unit/apps/dashboard/test_services.py
+++ b/tests/unit/apps/dashboard/test_services.py
@@ -1,4 +1,5 @@
 import json
+from decimal import Decimal
 from typing import Any, Callable, Dict, Optional
 
 from django.core.cache import cache
@@ -11,6 +12,7 @@
     DashboardDataCache,
     DashboardGlobalDataCache,
 )
+from hct_mis_api.apps.payment.models import Payment
 
 CACHE_CONFIG = [
     ("DashboardDataCache", DashboardDataCache, "test-area"),
@@ -29,7 +31,11 @@ def test_get_cache_key(cache_name: str, cache_class: Any, slug: str) -> None:
 @pytest.mark.django_db(databases=["default", "read_only"])
 def test_get_data_cache_hit(cache_name: str, cache_class: Any, slug: str) -> None:
     """Test get_data when data is found in the cache."""
-    cache.set(f"dashboard_data_{slug}", json.dumps({"test": f"{cache_name}_data"}), 60 * 60 * 24)
+    cache.set(
+        f"dashboard_data_{slug}",
+        json.dumps({"test": f"{cache_name}_data"}),
+        60 * 60 * 24,
+    )
     data: Optional[Dict[str, Any]] = cache_class.get_data(slug)
     assert data == {"test": f"{cache_name}_data"}
 
@@ -84,7 +90,8 @@ def test_refresh_data(
     cache_key = cache_class.get_cache_key(slug)
     cache.delete(cache_key)
 
-    _ = populate_dashboard_cache(afghanistan)
+    populate_dashboard_cache(afghanistan)
+
     refreshed_data = cache_class.refresh_data() if slug == "global" else cache_class.refresh_data(afghanistan.slug)
     assert refreshed_data is not None, "Refresh data returned None"
     assert len(refreshed_data) > 0, "No data returned by refresh"
@@ -99,3 +106,65 @@ def test_refresh_data(
         ), f"Expected optional fields {expected_optional_fields} are missing in {cache_name}: {item.keys()}"
     cached_data = cache.get(cache_key)
     assert cached_data is not None, "Data not cached"
+
+
+TEST_CASES = [
+    (
+        "delivered_quantity_usd prioritized",
+        {"delivered_quantity_usd": Decimal("100.0"), "entitlement_quantity_usd": Decimal("50.0")},
+        Decimal("500.0"),
+        DashboardDataCache,
+    ),
+    (
+        "entitlement_quantity_usd used when delivered_quantity_usd is null",
+        {"delivered_quantity_usd": None, "entitlement_quantity_usd": Decimal("50.0")},
+        Decimal("250.0"),
+        DashboardDataCache,
+    ),
+    (
+        "both fields null",
+        {"delivered_quantity_usd": None, "entitlement_quantity_usd": None},
+        Decimal("0.0"),
+        DashboardDataCache,
+    ),
+    (
+        "Pending status with null delivered_quantity_usd",
+        {"status": "Pending", "delivered_quantity_usd": None, "entitlement_quantity_usd": Decimal("50.0")},
+        Decimal("250.0"),
+        DashboardDataCache,
+    ),
+    (
+        "global cache prioritizes delivered_quantity_usd",
+        {"delivered_quantity_usd": Decimal("100.0"), "entitlement_quantity_usd": Decimal("50.0")},
+        Decimal("500.0"),
+        DashboardGlobalDataCache,
+    ),
+]
+
+
+@pytest.mark.parametrize(
+    "test_name, payment_updates, expected_total, cache_service",
+    TEST_CASES,
+    ids=[case[0] for case in TEST_CASES],
+)
+@pytest.mark.django_db(transaction=True, databases=["default", "read_only"])
+def test_dashboard_data_cache(
+    test_name: str,
+    payment_updates: Dict[str, Any],
+    expected_total: Decimal,
+    cache_service: Any,
+    populate_dashboard_cache: Callable,
+) -> None:
+    """Test DashboardDataCache and DashboardGlobalDataCache behavior."""
+    business_area = BusinessAreaFactory(slug="test-area")
+    household = populate_dashboard_cache(business_area)
+
+    Payment.objects.filter(household=household).update(**payment_updates)
+    if cache_service == DashboardDataCache:
+        result = cache_service.refresh_data(business_area.slug)
+    else:
+        result = cache_service.refresh_data()
+
+    total_usd = sum(Decimal(item["total_delivered_quantity_usd"]) for item in result)
+    assert len(result) > 0
+    assert total_usd == expected_total
diff --git a/tests/unit/fixtures/dashboard.py b/tests/unit/fixtures/dashboard.py
index 6dd0553a88d..c27dececd80 100644
--- a/tests/unit/fixtures/dashboard.py
+++ b/tests/unit/fixtures/dashboard.py
@@ -10,6 +10,7 @@
 from hct_mis_api.apps.household.fixtures import create_household
 from hct_mis_api.apps.household.models import Household
 from hct_mis_api.apps.payment.fixtures import PaymentFactory, PaymentPlanFactory
+from hct_mis_api.apps.program.fixtures import ProgramFactory
 
 
 class ModifiedPaymentFactory(PaymentFactory):
@@ -35,6 +36,8 @@ def _populate_dashboard_cache(afghanistan: BusinessAreaFactory) -> Household:
         Create household and related records
         """
         with transaction.atomic(using="default"):
+            program = ProgramFactory(business_area=afghanistan)
+
             household, _ = create_household(
                 household_args={
                     "business_area": afghanistan,
@@ -50,9 +53,10 @@ def _populate_dashboard_cache(afghanistan: BusinessAreaFactory) -> Household:
                     "male_age_group_18_59_disabled_count": 0,
                     "male_age_group_60_disabled_count": 1,
                     "admin1": AreaFactory(name="Kabul", area_type__name="Province", area_type__area_level=1),
+                    "program": program,
                 }
             )
-            ModifiedPaymentFactory.create_batch(5, household=household)
+            ModifiedPaymentFactory.create_batch(5, household=household, program=program)
 
         return household
 


Reviews:
user: login: mmaciekk
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-03-25T12:24:44Z
user: login: mmaciekk
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-03-25T12:25:11Z
user: login: mmaciekk
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-03-25T12:25:36Z
user: login: mmaciekk
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-03-25T12:25:57Z
user: login: mmaciekk
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-03-25T12:27:21Z
user: login: mmaciekk
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-03-25T12:27:51Z
user: login: mmaciekk
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-03-25T12:28:12Z
user: login: mmaciekk
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-03-25T12:28:40Z
user: login: mmaciekk
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-03-25T12:29:15Z
user: login: mmaciekk
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-03-25T12:29:30Z
user: login: mmaciekk
body: @srugano I suggest you revert the formatting and commit only the actual changes for this PR. It will be less prone to errors and much easier to review.
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-03-25T12:31:59Z
user: login: pavlo-mk
state: APPROVED
author_association: CONTRIBUTOR
submitted_at: 2025-03-25T12:35:14Z
user: login: domdinicola
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-03-25T12:48:57Z
user: login: domdinicola
state: APPROVED
author_association: CONTRIBUTOR
submitted_at: 2025-03-25T12:49:05Z
user: login: srugano
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-03-25T13:13:38Z

Review comments:
diff_hunk: @@ -6,4 +6,4 @@ repos:
         entry: docker compose -f development_tools/compose.yml run --no-TTY -i --rm backend sh -c 'black . --check && isort . --check-only && flake8 . && mypy .'
         language: system
         pass_filenames: false
-        verbose: true
path: .pre-commit-config.yaml
user: login: mmaciekk
body: just formatting?
created_at: 2025-03-25T12:24:43Z
author_association: CONTRIBUTOR
diff_hunk: @@ -30,7 +30,9 @@
 )
 
 finished_payment_plans = Count(
-    "parent__id", filter=Q(parent__payment_verification_plans__status="FINISHED"), distinct=True
+    "parent__id",
+    filter=Q(parent__payment_verification_plans__status="FINISHED"),
path: src/hct_mis_api/apps/dashboard/services.py
user: login: mmaciekk
body: just formatting?
created_at: 2025-03-25T12:25:11Z
author_association: CONTRIBUTOR
diff_hunk: @@ -91,17 +93,28 @@ def refresh_data(cls, business_area_slug: str) -> ReturnDict:
                 .filter(
                     business_area=area,
                     parent__status__in=["ACCEPTED", "FINISHED"],
+                    program__is_visible=True,
+                    parent__is_removed=False,
                     is_removed=False,
                     conflicted=False,
                 )  # noqa
-                .exclude(status__in=["Transaction Erroneous", "Not Distributed", "Force failed", "Manually Cancelled"])
+                .exclude(
+                    status__in=[
path: src/hct_mis_api/apps/dashboard/services.py
user: login: mmaciekk
body: just formatting?
created_at: 2025-03-25T12:25:36Z
author_association: CONTRIBUTOR
diff_hunk: @@ -91,17 +93,28 @@ def refresh_data(cls, business_area_slug: str) -> ReturnDict:
                 .filter(
                     business_area=area,
                     parent__status__in=["ACCEPTED", "FINISHED"],
+                    program__is_visible=True,
+                    parent__is_removed=False,
                     is_removed=False,
                     conflicted=False,
                 )  # noqa
-                .exclude(status__in=["Transaction Erroneous", "Not Distributed", "Force failed", "Manually Cancelled"])
+                .exclude(
+                    status__in=[
+                        "Transaction Erroneous",
+                        "Not Distributed",
+                        "Force failed",
+                        "Manually Cancelled",
+                    ]
+                )
                 .annotate(
                     year=ExtractYear(Coalesce("delivery_date", "entitlement_date", "status_date")),
                     month=ExtractMonth(Coalesce("delivery_date", "entitlement_date", "status_date")),
                     programs=Coalesce(F("household__program__name"), Value("Unknown program")),
                     sectors=Coalesce(F("household__program__sector"), Value("Unknown sector")),
                     admin1=Coalesce(
-                        F("household__admin1__name"), F("household__admin_area__name"), Value("Unknown admin area")
+                        F("household__admin1__name"),
+                        F("household__admin_area__name"),
path: src/hct_mis_api/apps/dashboard/services.py
user: login: mmaciekk
body: just formatting?
created_at: 2025-03-25T12:25:56Z
author_association: CONTRIBUTOR
diff_hunk: @@ -121,16 +134,30 @@ def refresh_data(cls, business_area_slug: str) -> ReturnDict:
                 .annotate(
                     total_usd=Sum(
                         Case(
-                            When(delivered_quantity_usd__isnull=False, then="delivered_quantity_usd"),
-                            When(entitlement_quantity_usd__isnull=False, then="entitlement_quantity_usd"),
+                            When(
+                                delivered_quantity_usd__isnull=False,
+                                then="delivered_quantity_usd",
path: src/hct_mis_api/apps/dashboard/services.py
user: login: mmaciekk
body: just formatting?
created_at: 2025-03-25T12:27:21Z
author_association: CONTRIBUTOR
diff_hunk: @@ -139,7 +166,11 @@ def refresh_data(cls, business_area_slug: str) -> ReturnDict:
                     individuals=Sum(Coalesce("household__size", Value(1))),
                     households=Count("household", distinct=True),
                     children_counts=Sum("household__children_count"),
-                    reconciled=Count("pk", distinct=False, filter=Q(payment_verifications__isnull=False)),
+                    reconciled=Count(
+                        "pk",
+                        distinct=False,
path: src/hct_mis_api/apps/dashboard/services.py
user: login: mmaciekk
body: just formatting?
created_at: 2025-03-25T12:27:51Z
author_association: CONTRIBUTOR
diff_hunk: @@ -184,7 +215,17 @@ def refresh_data(cls, business_area_slug: str) -> ReturnDict:
                 summary[key]["finished_payment_plans"] += item["finished_payment_plans"]
                 summary[key]["total_payment_plans"] += item["total_payment_plans"]
 
-            for (currency, year, month, program, sector, status, admin1, fsp, delivery_type), totals in summary.items():
+            for (
+                currency,
+                year,
+                month,
+                program,
+                sector,
path: src/hct_mis_api/apps/dashboard/services.py
user: login: mmaciekk
body: just formatting?
created_at: 2025-03-25T12:28:12Z
author_association: CONTRIBUTOR
diff_hunk: @@ -234,10 +275,19 @@ def refresh_data(cls, business_area_slug: str = "global") -> ReturnDict:
             )
             .filter(
                 parent__status__in=["ACCEPTED", "FINISHED"],
+                program__is_visible=True,
+                parent__is_removed=False,
                 is_removed=False,
                 conflicted=False,
             )  # noqa
-            .exclude(status__in=["Transaction Erroneous", "Not Distributed", "Force failed", "Manually Cancelled"])
+            .exclude(
+                status__in=[
+                    "Transaction Erroneous",
path: src/hct_mis_api/apps/dashboard/services.py
user: login: mmaciekk
body: just formatting?
created_at: 2025-03-25T12:28:40Z
author_association: CONTRIBUTOR
diff_hunk: @@ -256,8 +306,15 @@ def refresh_data(cls, business_area_slug: str = "global") -> ReturnDict:
             .annotate(
                 total_usd=Sum(
                     Case(
-                        When(delivered_quantity_usd__isnull=False, then="delivered_quantity_usd"),
-                        When(entitlement_quantity_usd__isnull=False, then="entitlement_quantity_usd"),
+                        When(
+                            delivered_quantity_usd__isnull=False,
path: src/hct_mis_api/apps/dashboard/services.py
user: login: mmaciekk
body: just formatting?
created_at: 2025-03-25T12:29:14Z
author_association: CONTRIBUTOR
diff_hunk: @@ -29,7 +31,11 @@ def test_get_cache_key(cache_name: str, cache_class: Any, slug: str) -> None:
 @pytest.mark.django_db(databases=["default", "read_only"])
 def test_get_data_cache_hit(cache_name: str, cache_class: Any, slug: str) -> None:
     """Test get_data when data is found in the cache."""
-    cache.set(f"dashboard_data_{slug}", json.dumps({"test": f"{cache_name}_data"}), 60 * 60 * 24)
+    cache.set(
+        f"dashboard_data_{slug}",
+        json.dumps({"test": f"{cache_name}_data"}),
path: tests/unit/apps/dashboard/test_services.py
user: login: mmaciekk
body: just formatting?
created_at: 2025-03-25T12:29:30Z
author_association: CONTRIBUTOR
diff_hunk: @@ -313,6 +313,9 @@ def login(browser: Chrome) -> Chrome:
     browser.find_element(By.ID, login).send_keys("superuser")
     browser.find_element(By.ID, password).send_keys("testtest2")
     browser.find_element(By.XPATH, loginButton).click()
+    from time import sleep
path: tests/selenium/conftest.py
user: login: domdinicola
body: import should be on top
created_at: 2025-03-25T12:48:57Z
author_association: CONTRIBUTOR
diff_hunk: @@ -313,6 +313,9 @@ def login(browser: Chrome) -> Chrome:
     browser.find_element(By.ID, login).send_keys("superuser")
     browser.find_element(By.ID, password).send_keys("testtest2")
     browser.find_element(By.XPATH, loginButton).click()
+    from time import sleep
path: tests/selenium/conftest.py
user: login: srugano
body: https://github.com/unicef/hope/pull/4679/commits/56b25667077dc8265c69ef2c1182e56ad1ed1cf4
created_at: 2025-03-25T13:13:38Z
author_association: CONTRIBUTOR

PR comments:
user: login: codecov[bot]
created_at: 2025-03-21T21:13:02Z
body: ## [Codecov](https://app.codecov.io/gh/unicef/hope/pull/4708?dropdown=coverage&src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=unicef) Report
All modified and coverable lines are covered by tests :white_check_mark:
> Project coverage is 85.86%. Comparing base [(`fda48a2`)](https://app.codecov.io/gh/unicef/hope/commit/fda48a2bf6a5fc16d35f9ff276f3b3cceafc0ca2?dropdown=coverage&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=unicef) to head [(`ac57423`)](https://app.codecov.io/gh/unicef/hope/commit/ac574230bcd5ece677a8b4d86a945f3600ccbd1d?dropdown=coverage&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=unicef).
> Report is 10 commits behind head on master.

<details><summary>Additional details and impacted files</summary>


```diff
@@           Coverage Diff           @@
##           master    #4708   +/-   ##
=======================================
  Coverage   85.85%   85.86%           
=======================================
  Files         548      548           
  Lines       36040    36040           
=======================================
+ Hits        30941    30944    +3     
+ Misses       5099     5096    -3     
```

| [Flag](https://app.codecov.io/gh/unicef/hope/pull/4708/flags?src=pr&el=flags&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=unicef) | Coverage Δ | |
|---|---|---|
| [e2e](https://app.codecov.io/gh/unicef/hope/pull/4708/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=unicef) | `60.85% <ø> (-1.31%)` | :arrow_down: |
| [unittests](https://app.codecov.io/gh/unicef/hope/pull/4708/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=unicef) | `84.34% <ø> (-0.15%)` | :arrow_down: |

Flags with carried forward coverage won't be shown. [Click here](https://docs.codecov.io/docs/carryforward-flags?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=unicef#carryforward-flags-in-the-pull-request-comment) to find out more.

</details>

[:umbrella: View full report in Codecov by Sentry](https://app.codecov.io/gh/unicef/hope/pull/4708?dropdown=coverage&src=pr&el=continue&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=unicef).   
:loudspeaker: Have feedback on the report? [Share it here](https://about.codecov.io/codecov-pr-comment-feedback/?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=unicef).

<details><summary> :rocket: New features to boost your workflow: </summary>

- :snowflake: [Test Analytics](https://docs.codecov.com/docs/test-analytics): Detect flaky tests, report on failures, and find test suite problems.
- :package: [JS Bundle Analysis](https://docs.codecov.com/docs/javascript-bundle-analysis): Save yourself from yourself by tracking and limiting bundle sizes in JS merges.
</details>
author_association: NONE
user: login: srugano
created_at: 2025-03-25T13:23:36Z
body: > @srugano I suggest you revert the formatting and commit only the actual changes for this PR. It will be less prone to errors and much easier to review.

Updated @mmaciekk 
author_association: CONTRIBUTOR