Repository: Second-Hand-Friends/kleinanzeigen-bot
Pull request #483: feat: add support for multiple matching shipping options
Author: login: taberruu
Labels: name: enhancement
description: New feature or request
Created: 2025-04-26T13:39:07Z | Merged: 2025-04-29T19:02:09Z

Add functionality to include all shipping options of the same package size when `include_all_matching_shipping_options` is enabled

## ℹ️ Description
This PR adds functionality to store all shipping options of the same package size when downloading ads, instead of just the one matching the exact price. This is useful when multiple shipping options are available for the same package size (e.g., DHL and Hermes options for small packages). 
Additionally, it allows excluding specific shipping options (e.g., when certain carriers are not available in a region).

- Link to the related issue(s): N/A
- The motivation for this change is to provide more flexibility in shipping options when downloading ads, allowing users to:
  - See all available shipping options for a given package size rather than just the one matching the exact price
  - Exclude specific shipping options that are not available in their region (e.g., DHL not available in certain areas)

## 📋 Changes Summary

- Added new configuration options under `download` section:
  - `include_all_matching_shipping_options`: Controls whether all shipping options of the same package size should be stored
  - `excluded_shipping_options`: Allows excluding specific shipping options (e.g., when certain carriers are not available in a region)
- Modified shipping option extraction logic to:
  - Consider package size when matching shipping options
  - Support storing multiple shipping options of the same package size
  - Handle excluded shipping options (e.g., if DHL_2 is excluded, it won't be stored even if it matches the package size)

### ⚙️ Type of Change
Select the type(s) of change(s) included in this pull request:
- [ ] 🐞 Bug fix (non-breaking change which fixes an issue)
- [x] ✨ New feature (adds new functionality without breaking existing usage)
- [ ] 💥 Breaking change (changes that might break existing user setups, scripts, or configurations)


## ✅ Checklist
Before requesting a review, confirm the following:
- [x] I have reviewed my changes to ensure they meet the project's standards.
- [x] I have tested my changes and ensured that all tests pass  (`pdm run test`).
- [x] I have verified that linting passes (`pdm run lint`).
- [x] I have run security scans and addressed any identified issues (`pdm run audit`).
- [x] I have updated documentation where necessary.

By submitting this pull request, I confirm that you can use, modify, copy, and redistribute this contribution, under the terms of your choice.



Diff:
diff --git a/src/kleinanzeigen_bot/extract.py b/src/kleinanzeigen_bot/extract.py
index 82e4daa8..52b7bf94 100644
--- a/src/kleinanzeigen_bot/extract.py
+++ b/src/kleinanzeigen_bot/extract.py
@@ -422,11 +422,6 @@ async def _extract_shipping_info_from_ad_page(self) -> tuple[str, float | None,
                     (await self.web_request("https://gateway.kleinanzeigen.de/postad/api/v1/shipping-options?posterType=PRIVATE"))
                     ["content"])["data"]["shippingOptionsResponse"]["options"]
 
-                internal_shipping_opt = [x for x in shipping_costs if x["priceInEuroCent"] == ship_costs * 100]
-
-                if not internal_shipping_opt:
-                    return "NOT_APPLICABLE", ship_costs, shipping_options
-
                 # map to internal shipping identifiers used by kleinanzeigen-bot
                 shipping_option_mapping = {
                     "DHL_001": "DHL_2",
@@ -440,11 +435,41 @@ async def _extract_shipping_info_from_ad_page(self) -> tuple[str, float | None,
                     "HERMES_004": "Hermes_L"
                 }
 
-                shipping_option = shipping_option_mapping.get(internal_shipping_opt[0]["id"])
-                if not shipping_option:
-                    return "NOT_APPLICABLE", ship_costs, shipping_options
+                # Convert Euro to cents and round to nearest integer
+                price_in_cent = round(ship_costs * 100)
+
+                # Get excluded shipping options from config
+                excluded_options = self.config.get("download", {}).get("excluded_shipping_options", [])
+
+                # If include_all_matching_shipping_options is enabled, get all options for the same package size
+                if self.config.get("download", {}).get("include_all_matching_shipping_options", False):
+                    # Find all options with the same price to determine the package size
+                    matching_options = [opt for opt in shipping_costs if opt["priceInEuroCent"] == price_in_cent]
+                    if not matching_options:
+                        return "NOT_APPLICABLE", ship_costs, shipping_options
+
+                    # Use the package size of the first matching option
+                    matching_size = matching_options[0]["packageSize"]
+
+                    # Get all options of the same size
+                    shipping_options = [
+                        shipping_option_mapping[opt["id"]]
+                        for opt in shipping_costs
+                        if opt["packageSize"] == matching_size
+                        and opt["id"] in shipping_option_mapping
+                        and shipping_option_mapping[opt["id"]] not in excluded_options
+                    ]
+                else:
+                    # Only use the matching option if it's not excluded
+                    matching_option = next((x for x in shipping_costs if x["priceInEuroCent"] == price_in_cent), None)
+                    if not matching_option:
+                        return "NOT_APPLICABLE", ship_costs, shipping_options
+
+                    shipping_option = shipping_option_mapping.get(matching_option["id"])
+                    if not shipping_option or shipping_option in excluded_options:
+                        return "NOT_APPLICABLE", ship_costs, shipping_options
+                    shipping_options = [shipping_option]
 
-                shipping_options = [shipping_option]
         except TimeoutError:  # no pricing box -> no shipping given
             ship_type = "NOT_APPLICABLE"
 
diff --git a/src/kleinanzeigen_bot/resources/config_defaults.yaml b/src/kleinanzeigen_bot/resources/config_defaults.yaml
index 8097543a..3455e588 100644
--- a/src/kleinanzeigen_bot/resources/config_defaults.yaml
+++ b/src/kleinanzeigen_bot/resources/config_defaults.yaml
@@ -26,6 +26,12 @@ ad_defaults:
 #   Jobs > Praktika: 102/125
 categories: {}
 
+download:
+  # if true, all shipping options matching the package size will be included
+  include_all_matching_shipping_options: false
+  # list of shipping options to exclude, e.g. ["DHL_2", "DHL_5"]
+  excluded_shipping_options: []
+
 publishing:
   delete_old_ads: "AFTER_PUBLISH"  # one of: AFTER_PUBLISH, BEFORE_PUBLISH, NEVER
   delete_old_ads_by_title: true  # only works if delete_old_ads is set to BEFORE_PUBLISH
diff --git a/tests/unit/test_extract.py b/tests/unit/test_extract.py
index f4b88b21..b382de58 100644
--- a/tests/unit/test_extract.py
+++ b/tests/unit/test_extract.py
@@ -107,7 +107,7 @@ async def test_extract_shipping_info(
                     "data": {
                         "shippingOptionsResponse": {
                             "options": [
-                                {"id": "DHL_001", "priceInEuroCent": int(expected_cost * 100)}
+                                {"id": "DHL_001", "priceInEuroCent": int(expected_cost * 100), "packageSize": "SMALL"}
                             ]
                         }
                     }
@@ -132,7 +132,7 @@ async def test_extract_shipping_info_with_options(self, test_extractor:AdExtract
                 "data": {
                     "shippingOptionsResponse": {
                         "options": [
-                            {"id": "DHL_001", "priceInEuroCent": 549}
+                            {"id": "DHL_001", "priceInEuroCent": 549, "packageSize": "SMALL"}
                         ]
                     }
                 }
@@ -149,6 +149,109 @@ async def test_extract_shipping_info_with_options(self, test_extractor:AdExtract
             assert costs == 5.49
             assert options == ["DHL_2"]
 
+    @pytest.mark.asyncio
+    # pylint: disable=protected-access
+    async def test_extract_shipping_info_with_all_matching_options(self, test_extractor: AdExtractor) -> None:
+        """Test shipping info extraction with all matching options enabled."""
+        shipping_response = {
+            "content": json.dumps({
+                "data": {
+                    "shippingOptionsResponse": {
+                        "options": [
+                            {"id": "HERMES_001", "priceInEuroCent": 489, "packageSize": "SMALL"},
+                            {"id": "HERMES_002", "priceInEuroCent": 549, "packageSize": "SMALL"},
+                            {"id": "DHL_001", "priceInEuroCent": 619, "packageSize": "SMALL"}
+                        ]
+                    }
+                }
+            })
+        }
+
+        # Enable all matching options in config
+        test_extractor.config["download"] = {"include_all_matching_shipping_options": True}
+
+        with patch.object(test_extractor, "page", MagicMock()), \
+                patch.object(test_extractor, "web_text", new_callable = AsyncMock, return_value = "+ Versand ab 4,89 €"), \
+                patch.object(test_extractor, "web_request", new_callable = AsyncMock, return_value = shipping_response):
+
+            shipping_type, costs, options = await test_extractor._extract_shipping_info_from_ad_page()
+
+            assert shipping_type == "SHIPPING"
+            assert costs == 4.89
+            if options is not None:
+                assert sorted(options) == ["DHL_2", "Hermes_Päckchen", "Hermes_S"]
+            else:
+                assert options is None
+
+    @pytest.mark.asyncio
+    # pylint: disable=protected-access
+    async def test_extract_shipping_info_with_excluded_options(self, test_extractor: AdExtractor) -> None:
+        """Test shipping info extraction with excluded options."""
+        shipping_response = {
+            "content": json.dumps({
+                "data": {
+                    "shippingOptionsResponse": {
+                        "options": [
+                            {"id": "HERMES_001", "priceInEuroCent": 489, "packageSize": "SMALL"},
+                            {"id": "HERMES_002", "priceInEuroCent": 549, "packageSize": "SMALL"},
+                            {"id": "DHL_001", "priceInEuroCent": 619, "packageSize": "SMALL"}
+                        ]
+                    }
+                }
+            })
+        }
+
+        # Enable all matching options and exclude DHL in config
+        test_extractor.config["download"] = {
+            "include_all_matching_shipping_options": True,
+            "excluded_shipping_options": ["DHL_2"]
+        }
+
+        with patch.object(test_extractor, "page", MagicMock()), \
+                patch.object(test_extractor, "web_text", new_callable = AsyncMock, return_value = "+ Versand ab 4,89 €"), \
+                patch.object(test_extractor, "web_request", new_callable = AsyncMock, return_value = shipping_response):
+
+            shipping_type, costs, options = await test_extractor._extract_shipping_info_from_ad_page()
+
+            assert shipping_type == "SHIPPING"
+            assert costs == 4.89
+            if options is not None:
+                assert sorted(options) == ["Hermes_Päckchen", "Hermes_S"]
+            else:
+                assert options is None
+
+    @pytest.mark.asyncio
+    # pylint: disable=protected-access
+    async def test_extract_shipping_info_with_excluded_matching_option(self, test_extractor: AdExtractor) -> None:
+        """Test shipping info extraction when the matching option is excluded."""
+        shipping_response = {
+            "content": json.dumps({
+                "data": {
+                    "shippingOptionsResponse": {
+                        "options": [
+                            {"id": "HERMES_001", "priceInEuroCent": 489, "packageSize": "SMALL"},
+                            {"id": "HERMES_002", "priceInEuroCent": 549, "packageSize": "SMALL"}
+                        ]
+                    }
+                }
+            })
+        }
+
+        # Exclude the matching option
+        test_extractor.config["download"] = {
+            "excluded_shipping_options": ["Hermes_Päckchen"]
+        }
+
+        with patch.object(test_extractor, "page", MagicMock()), \
+                patch.object(test_extractor, "web_text", new_callable = AsyncMock, return_value = "+ Versand ab 4,89 €"), \
+                patch.object(test_extractor, "web_request", new_callable = AsyncMock, return_value = shipping_response):
+
+            shipping_type, costs, options = await test_extractor._extract_shipping_info_from_ad_page()
+
+            assert shipping_type == "NOT_APPLICABLE"
+            assert costs == 4.89
+            assert options is None
+
 
 class TestAdExtractorNavigation:
     """Tests for navigation related functionality."""


Reviews:
user: login: Heavenfighter
body: Nice and clean PR. The code works as intended.

state: DISMISSED
author_association: CONTRIBUTOR
submitted_at: 2025-04-28T05:45:25Z
user: login: sebthom
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-04-29T18:36:39Z
user: login: sebthom
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-04-29T18:38:08Z
user: login: taberruu
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-04-29T18:49:41Z
user: login: taberruu
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-04-29T18:49:46Z
user: login: sebthom
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-04-29T18:51:59Z

Review comments:
diff_hunk: @@ -26,6 +26,12 @@ ad_defaults:
 #   Jobs > Praktika: 102/125
 categories: {}
 
+download:
+  # if true, all shipping options matching the size will be stored
path: src/kleinanzeigen_bot/resources/config_defaults.yaml
user: login: sebthom
body: can you replace "size" with "package size"?
created_at: 2025-04-29T18:36:38Z
author_association: CONTRIBUTOR
diff_hunk: @@ -26,6 +26,12 @@ ad_defaults:
 #   Jobs > Praktika: 102/125
 categories: {}
 
+download:
+  # if true, all shipping options matching the size will be stored
+  store_all_matching_shipping_options: false
path: src/kleinanzeigen_bot/resources/config_defaults.yaml
user: login: sebthom
body: WDYT about renaming "store_" to "include_"?
created_at: 2025-04-29T18:38:08Z
author_association: CONTRIBUTOR
diff_hunk: @@ -26,6 +26,12 @@ ad_defaults:
 #   Jobs > Praktika: 102/125
 categories: {}
 
+download:
+  # if true, all shipping options matching the size will be stored
path: src/kleinanzeigen_bot/resources/config_defaults.yaml
user: login: taberruu
body: Thanks for the suggestion! I've updated the comment to be more precise about package size.
created_at: 2025-04-29T18:49:41Z
author_association: CONTRIBUTOR
diff_hunk: @@ -26,6 +26,12 @@ ad_defaults:
 #   Jobs > Praktika: 102/125
 categories: {}
 
+download:
+  # if true, all shipping options matching the size will be stored
+  store_all_matching_shipping_options: false
path: src/kleinanzeigen_bot/resources/config_defaults.yaml
user: login: taberruu
body: Good catch! I've renamed the option to better reflect that it includes matching shipping options.
created_at: 2025-04-29T18:49:45Z
author_association: CONTRIBUTOR
diff_hunk: @@ -26,6 +26,12 @@ ad_defaults:
 #   Jobs > Praktika: 102/125
 categories: {}
 
+download:
+  # list of shipping options to exclude, e.g. ["DHL_2", "DHL_5"]
path: src/kleinanzeigen_bot/resources/config_defaults.yaml
user: login: sebthom
body: the comment seems to have moved to the wrong line
created_at: 2025-04-29T18:51:59Z
author_association: CONTRIBUTOR

PR comments:
user: login: sebthom
created_at: 2025-04-28T11:32:58Z
body: @benjidea would you please rebase your PR. There is a merge conflict now.
author_association: CONTRIBUTOR
user: login: sebthom
created_at: 2025-04-29T19:02:15Z
body: Thanks!
author_association: CONTRIBUTOR