Repository: swisstopo/swissgeol-boreholes-dataextraction
Pull request #188: check for intersections between material descriptions and sidebars
Author: login: TicaGit
Created: 2025-04-25T09:05:43Z | Merged: 2025-04-30T13:37:03Z

## This PR resolves #130 

This issue was not reproducible anymore, the overlapping inside deepwell 3EK5LEG800_bp_19940601_Arsch93-18.pdf was already resolved by PR#174, and no similar problems existed in the datasets. I implemented a quick solution anyway, as this problem could appear in some other pdf unseen yet.

## This PR also resolves #153 

Some additional checks are applied to filter out material descriptions with sidebars before validating them as actual boreholes.

The first check removes any pair that contains no depth information (i.e.all depth values are None). This affects the following files:

GeoQuat
- False positives removed (good): 14225.pdf, A11406.pdf, 1488.pdf, 14273.pdf
- True positive removed (bad): A1304.pdf: According to the ground truth, this file contains three boreholes. One is correctly detected. The second borehole is removed at this stage because its depth column was not detected during extraction, leaving it without any depth values (see image below). The third borehole is not detected at all.

Zurich
- False positives removed (good): 695265010-bp.pdf : A column of incorrect numbers is removed. The main borehole isn’t detected in this file, as it is rotated).

Overall, this check seems justified. The only true positive removed is due to an error in the extraction pipeline, which fails to detect a depth column, rather than a flaw in the check itself.

---

The second check finds duplicates in the pairs. It affects the following files:
GeoQuat
- False positives removed (good): 8366.pdf, 13080.pdf, 7317.pdf

This check is only beneficial.

---

Among the list provided in issue #153, the following document remains false positives and here the reason why:
- 5872.pdf: thickness of layers column is detected instead of depth.
- A263.pdf : a series of explanation containing diameter numbers is detected as Sidebar (because it contains 1), ...5) ).
- 7703.pdf: compactness column is detected as a borehole.
- 684252058-bp.pdf: water infiltration is detected instead of depth, this breaks the borehole into 2 incorrect parts.
- 268124125-bp.pdf: humidity column detected

All those issue are related to incorrect depth column extraction and not to duplication, so issue #153 is effectively resolved.

Also note that:
- 9394.pdf was already resolved by  PR#174

<img width="574" alt="image" src="https://github.com/user-attachments/assets/5d067ca3-828b-437c-8e1b-87e5cb5c8f4b" />

Linked issue #130: Dont allow MaterialDescriptionRectWithSidebar to overlap
Currently, in extract.py (process_page()), we remove material description-sidebar pairs (material_descriptions_sidebar_pairs) if their material description bounding box (material_description_rect) intersect with those of another pair. However, we do not check whether the sidebar of one pair intersects with the material description rect of another pair.

We should modify the logic to also prevent sidebars from overlapping with other material description rects to ensure separation. 
Example of where we recieve 2 MaterialDescriptionRectWithSidebars for deepwell 3EK5LEG800_bp_19940601_Arsch93-18.pdf

![Image](https://github.com/user-attachments/assets/853e00e1-d4a6-4c2a-b4da-ee3b155cf753)

Diff:
diff --git a/config/matching_params.yml b/config/matching_params.yml
index f499e3ec..25bbead2 100644
--- a/config/matching_params.yml
+++ b/config/matching_params.yml
@@ -2,6 +2,7 @@
 block_line_ratio: 0.20
 min_block_clearance: 10
 left_line_length_threshold: 7
+duplicate_similarity_threshold: 0.5
 layer_identifier_acceptance_ratio: 0.5
 img_template_probability_threshold: 0.62
 
diff --git a/src/extraction/annotations/draw.py b/src/extraction/annotations/draw.py
index 2767d8bb..d31b1061 100644
--- a/src/extraction/annotations/draw.py
+++ b/src/extraction/annotations/draw.py
@@ -97,18 +97,19 @@ def draw_predictions(
                             page.derotation_matrix,
                             [layer for layer in bh_layers.layers if layer.material_description.page == page_number],
                         )
-                        shape.commit()  # Commit all the drawing operations to the page
 
-                        tmp_file_path = out_directory / f"{filename}_page{page_number}.png"
-                        pymupdf.utils.get_pixmap(page, matrix=pymupdf.Matrix(2, 2), clip=page.rect).save(tmp_file_path)
+                    shape.commit()  # Commit all the drawing operations to the page
 
-                        if mlflow_tracking:  # This is only executed if MLFlow tracking is enabled
-                            try:
-                                import mlflow
+                    tmp_file_path = out_directory / f"{filename}_page{page_number}.png"
+                    pymupdf.utils.get_pixmap(page, matrix=pymupdf.Matrix(2, 2), clip=page.rect).save(tmp_file_path)
 
-                                mlflow.log_artifact(tmp_file_path, artifact_path="pages")
-                            except NameError:
-                                logger.warning("MLFlow could not be imported. Skipping logging of artifact.")
+                    if mlflow_tracking:  # This is only executed if MLFlow tracking is enabled
+                        try:
+                            import mlflow
+
+                            mlflow.log_artifact(tmp_file_path, artifact_path="pages")
+                        except NameError:
+                            logger.warning("MLFlow could not be imported. Skipping logging of artifact.")
 
         except (FileNotFoundError, pymupdf.FileDataError) as e:
             logger.error("Error opening file %s: %s", filename, e)
diff --git a/src/extraction/features/extract.py b/src/extraction/features/extract.py
index 6ec79771..1b4c5c1d 100644
--- a/src/extraction/features/extract.py
+++ b/src/extraction/features/extract.py
@@ -12,6 +12,7 @@
     MaterialDescriptionRectWithSidebar,
     PageBoundingBoxes,
 )
+from extraction.features.stratigraphy.sidebar.classes.layer_identifier_sidebar import LayerIdentifierSidebar
 from extraction.features.stratigraphy.sidebar.classes.sidebar import (
     Sidebar,
     SidebarNoise,
@@ -91,25 +92,129 @@ def process_page(self) -> list[ExtractedBorehole]:
             pair for pair in material_descriptions_sidebar_pairs if pair.score_match >= 0
         ]
 
-        to_delete = []
-        for i, pair in enumerate(material_descriptions_sidebar_pairs):
-            if any(
-                pair.material_description_rect.intersects(other_pair.material_description_rect)
-                for other_pair in material_descriptions_sidebar_pairs[i + 1 :]
-            ):
-                to_delete.append(i)
-
+        # remove pairs that have any of their elements (sidebar, material description) intersecting with others.
+        to_delete = self._find_intersecting_indices(material_descriptions_sidebar_pairs)
         filtered_pairs = [
             item for index, item in enumerate(material_descriptions_sidebar_pairs) if index not in to_delete
         ]
 
+        # remove pairs with no sidebar if there is more than one pair.
+        to_delete = (
+            [] if len(filtered_pairs) <= 1 else [i for i, pair in enumerate(filtered_pairs) if not pair.sidebar]
+        )
+
+        filtered_pairs = [item for index, item in enumerate(filtered_pairs) if index not in to_delete]
+
+        # remove pairs that are likely duplicates of others.
+        to_delete = self._find_duplicated_pairs_indices(filtered_pairs)
+        non_duplicated_pairs = [item for index, item in enumerate(filtered_pairs) if index not in to_delete]
+
         # We order the boreholes with the highest score first. When one borehole is actually present in the ground
         # truth, but more than one are detected, we want the most correct to be assigned
-        # TODO we actually should try to merge similar extracted boreholes to keep the most information possible.
         return [
             self._create_borehole_from_pair(pair)
-            for pair in sorted(filtered_pairs, key=lambda pair: pair.score_match, reverse=True)
+            for pair in sorted(non_duplicated_pairs, key=lambda pair: pair.score_match, reverse=True)
+        ]
+
+    def _find_intersecting_indices(
+        self, material_descriptions_sidebar_pairs: list[MaterialDescriptionRectWithSidebar]
+    ) -> list[int]:
+        """Identifies overlapping material descriptions or sidebars.
+
+        This function scans through all material description/sidebar pairs and returns a list of indices
+        that should be removed due to overlaps. If an intersection is found between two elements, only
+        the first (lower-indexed) element is marked for deletion.
+
+        Args:
+            material_descriptions_sidebar_pairs (list[MaterialDescriptionRectWithSidebar]): a list of pairs consisting
+                of a material description rectangle and an optional sidebar.
+
+        Returns:
+            list[int]: The indices of elements that overlap with others and should be removed.
+        """
+        to_delete = []
+
+        def intersects_any(rect, others):
+            return any(rect.intersects(other) for other in others if other is not None)
+
+        for i, pair in enumerate(material_descriptions_sidebar_pairs):
+            mat_rect = pair.material_description_rect
+            sidebar_rect = pair.sidebar.rect() if pair.sidebar else None
+
+            # Build the list of other rectangles
+            remaining_pairs = material_descriptions_sidebar_pairs[i + 1 :]
+            other_mat_rects = [p.material_description_rect for p in remaining_pairs]
+            other_sidebar_rects = [p.sidebar.rect() if p.sidebar else None for p in remaining_pairs]
+
+            # Check all conditions
+            if (
+                intersects_any(mat_rect, other_mat_rects)
+                or intersects_any(mat_rect, other_sidebar_rects)
+                or (sidebar_rect and intersects_any(sidebar_rect, other_mat_rects))
+                or (sidebar_rect and intersects_any(sidebar_rect, other_sidebar_rects))
+            ):
+                to_delete.append(i)
+        return to_delete
+
+    def _find_duplicated_pairs_indices(self, filtered_pairs: list[MaterialDescriptionRectWithSidebar]) -> list[int]:
+        """Identify indices of pairs that are likely duplicates.
+
+        This is done because the informations about the same borehole could be represented in multiple ways.
+        We only want to keep multiple pairs if they belong to different boreholes in the given pdf page.
+        If duplication is found, we delete the first element, as the pairs are sorted from lowest score first.
+
+        Args:
+            filtered_pairs (list[MaterialDescriptionRectWithSidebar]): The filtered pairs.
+
+        Returns:
+            list[int]: A list containing the indexes of pairs that are duplicated and that should be deleted.
+        """
+        if len(filtered_pairs) <= 1:
+            # only one pair, can't be a duplicate
+            return []
+
+        all_interval_lists = [
+            [interval.depth_interval for interval in self._get_interval_block_pairs(pair)] for pair in filtered_pairs
         ]
+        no_depths = any([all(interval is None for interval in interval_list) for interval_list in all_interval_lists])
+        if no_depths:
+            # if a sidebar does not have depths associate with it, we check if all the sidebars contains layer
+            # identifiers like a), b), ...
+            if all([isinstance(p.sidebar, LayerIdentifierSidebar) for p in filtered_pairs]):
+                # We create a set with all the layer identifier entries of each sidebars, we will check for
+                # duplicates using those elements
+                all_element_sets = [set([entry.value for entry in p.sidebar.entries]) for p in filtered_pairs]
+            else:
+                return []  # otherwise, we don't delete anything
+        else:
+            # If all the sidebars have depths associate with them, we create sets with all the depths appearing in
+            # each interval list and check for duplicates using those elements.
+            all_element_sets = []
+            for interval_list in all_interval_lists:
+                depth_set = set()
+                for interval in interval_list:
+                    if interval is None:
+                        continue
+                    if interval.start:
+                        depth_set.add(interval.start.value)
+                    if interval.end:
+                        depth_set.add(interval.end.value)
+                all_element_sets.append(depth_set)
+
+        to_delete = []
+        # Compare the element sets (depths or layer identifiers) and compute their overlap ratio.
+        # If two sets share many of the same elements, they are likely duplicates.
+        for i, depth in enumerate(all_element_sets):
+            for other_depth in all_element_sets[i + 1 :]:
+                intersection_len = len(depth & other_depth)
+                min_len = min(len(depth), len(other_depth))
+
+                similarity_score = intersection_len / min_len if min_len != 0 else 0
+
+                if similarity_score > self.params["duplicate_similarity_threshold"]:
+                    to_delete.append(i)
+
+        return to_delete
 
     def _create_borehole_from_pair(self, pair: MaterialDescriptionRectWithSidebar) -> ExtractedBorehole:
         bounding_boxes = PageBoundingBoxes.from_material_description_rect_with_sidebar(pair, self.page_number)


Reviews:
user: login: copilot-pull-request-reviewer[bot]
body: ## Pull Request Overview

This PR implements extra filtering steps in the extraction pipeline to handle potential intersections between material description rectangles and sidebars, as well as to filter out pairs without depth information and to remove duplicate pairs. It also updates the configuration with a duplicate similarity threshold.
- Updated intersection filtering logic with a dedicated helper method.
- Added checks to remove pairs with no depth data and to remove likely duplicate pairs.
- Updated matching configuration by adding a new parameter.

### Reviewed Changes

Copilot reviewed 2 out of 2 changed files in this pull request and generated 2 comments.

| File                                       | Description                                                 |
| ------------------------------------------ | ----------------------------------------------------------- |
| src/extraction/features/extract.py         | Added helper methods for intersection, no-depth, and duplicate checks, and updated the borehole extraction logic. |
| config/matching_params.yml                 | Added duplicate_similarity_threshold parameter.           |




state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-04-25T09:11:54Z
user: login: stijnvermeeren-swisstopo
state: COMMENTED
author_association: COLLABORATOR
submitted_at: 2025-04-25T12:55:49Z
user: login: copilot-pull-request-reviewer[bot]
body: ## Pull Request Overview

This PR resolves issues related to overlapping material description and sidebar pairs, and improves the filtering of pairs with missing depth information or duplicate detections. Key changes include refactoring deletion logic by introducing helper functions (_find_intersecting_indices, _find_no_depths_indices, _find_duplicated_pairs_indices) in extract.py, a minor update in the drawing commit in draw.py, and the addition of a duplicate similarity threshold in matching_params.yml.

### Reviewed Changes

Copilot reviewed 3 out of 3 changed files in this pull request and generated 2 comments.

| File                                       | Description                                                       |
|--------------------------------------------|-------------------------------------------------------------------|
| src/extraction/features/extract.py         | Refactored overlapping, no-depth, and duplicate pair filtering logic. |
| src/extraction/annotations/draw.py         | Minor update to ensure drawing operations are committed properly. |
| config/matching_params.yml                   | Added duplicate_similarity_threshold configuration.               |




state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-04-28T05:45:35Z
user: login: copilot-pull-request-reviewer[bot]
body: ## Pull Request Overview

This PR addresses issues related to borehole extraction by improving overlap detection and duplicate filtering between material descriptions and sidebars. Key changes include:
- Replacing the inlined intersection check with a dedicated _find_intersecting_indices function.
- Adding additional filtering functions (_find_no_depths_indices and _find_duplicated_pairs_indices) to remove pairs with no depth values or those likely duplicated.
- Updating matching parameters in config/matching_params.yml to include duplicate_similarity_threshold.

### Reviewed Changes

Copilot reviewed 3 out of 3 changed files in this pull request and generated 1 comment.

| File                                      | Description                                                                                        |
| ----------------------------------------- | -------------------------------------------------------------------------------------------------- |
| src/extraction/features/extract.py        | Refactored pair filtering by adding intersection, no-depth, and duplicate detection helper methods. |
| src/extraction/annotations/draw.py         | Minor adjustments within drawing functions, with no functional changes.                         |
| config/matching_params.yml                 | Added a new parameter for duplicate similarity threshold to support duplicate removal logic.    |




state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-04-28T05:49:02Z
user: login: TicaGit
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-04-28T05:53:55Z
user: login: stijnvermeeren-swisstopo
state: CHANGES_REQUESTED
author_association: COLLABORATOR
submitted_at: 2025-04-28T09:41:09Z
user: login: TicaGit
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-04-28T16:09:47Z
user: login: stijnvermeeren-swisstopo
body: I like the new behaviour, it's exactly what I intended with my previous comment. As you said, the implementation is not entirely satisfactory yet. There's a lot of code that only covers a small number of known input PDFs, and there's a duplicated call to the `_get_interval_block_pairs` method, which is somewhat confusing and inefficient. However, when thinking about how to solve it in a better way, I did not see an easy fix, but rather I think it will involve some more fundamental changes to how we find/identify boreholes in the first place, and that's certainly out-of-scope for this PR. For that reason, I created a new issue https://github.com/swisstopo/swissgeol-boreholes-dataextraction/issues/195 and I'm happy to merge this as a temporary solution in the meantime.
state: APPROVED
author_association: COLLABORATOR
submitted_at: 2025-04-30T12:21:05Z

Review comments:
diff_hunk: @@ -91,26 +91,143 @@ def process_page(self) -> list[ExtractedBorehole]:
             pair for pair in material_descriptions_sidebar_pairs if pair.score_match >= 0
         ]
 
-        to_delete = []
-        for i, pair in enumerate(material_descriptions_sidebar_pairs):
-            if any(
-                pair.material_description_rect.intersects(other_pair.material_description_rect)
-                for other_pair in material_descriptions_sidebar_pairs[i + 1 :]
-            ):
-                to_delete.append(i)
-
+        # remove pairs that have any of their elements (sidebar, material description) intersecting with others.
+        to_delete = self._find_intersecting_indices(material_descriptions_sidebar_pairs)
         filtered_pairs = [
             item for index, item in enumerate(material_descriptions_sidebar_pairs) if index not in to_delete
         ]
 
+        # remove pairs that have all depths equal to None (only if they is more than one pair).
+        to_delete = self._find_no_depths_indices(filtered_pairs)
+        filtered_pairs = [item for index, item in enumerate(filtered_pairs) if index not in to_delete]
+
+        # remove pairs that are likelly duplicates of others.
+        to_delete = self._find_duplicated_pairs_indices(filtered_pairs)
path: src/extraction/features/extract.py
user: login: Copilot
body: The word 'likelly' is misspelled; consider replacing it with 'likely'.
```suggestion
        # remove pairs that are likely duplicates of others.
```
created_at: 2025-04-25T09:11:53Z
author_association: CONTRIBUTOR
diff_hunk: @@ -91,26 +91,143 @@ def process_page(self) -> list[ExtractedBorehole]:
             pair for pair in material_descriptions_sidebar_pairs if pair.score_match >= 0
         ]
 
-        to_delete = []
-        for i, pair in enumerate(material_descriptions_sidebar_pairs):
-            if any(
-                pair.material_description_rect.intersects(other_pair.material_description_rect)
-                for other_pair in material_descriptions_sidebar_pairs[i + 1 :]
-            ):
-                to_delete.append(i)
-
+        # remove pairs that have any of their elements (sidebar, material description) intersecting with others.
+        to_delete = self._find_intersecting_indices(material_descriptions_sidebar_pairs)
         filtered_pairs = [
             item for index, item in enumerate(material_descriptions_sidebar_pairs) if index not in to_delete
         ]
 
+        # remove pairs that have all depths equal to None (only if they is more than one pair).
+        to_delete = self._find_no_depths_indices(filtered_pairs)
path: src/extraction/features/extract.py
user: login: Copilot
body: The phrase 'if they is more than one pair' is grammatically incorrect; consider changing it to 'if there is more than one pair'.
```suggestion
        # remove pairs that have all depths equal to None (only if there is more than one pair).
```
created_at: 2025-04-25T09:11:54Z
author_association: CONTRIBUTOR
diff_hunk: @@ -91,26 +91,141 @@ def process_page(self) -> list[ExtractedBorehole]:
             pair for pair in material_descriptions_sidebar_pairs if pair.score_match >= 0
         ]
 
-        to_delete = []
-        for i, pair in enumerate(material_descriptions_sidebar_pairs):
-            if any(
-                pair.material_description_rect.intersects(other_pair.material_description_rect)
-                for other_pair in material_descriptions_sidebar_pairs[i + 1 :]
-            ):
-                to_delete.append(i)
-
+        # remove pairs that have any of their elements (sidebar, material description) intersecting with others.
+        to_delete = self._find_intersecting_indices(material_descriptions_sidebar_pairs)
         filtered_pairs = [
             item for index, item in enumerate(material_descriptions_sidebar_pairs) if index not in to_delete
         ]
 
+        # remove pairs that have all depths equal to None (only if there is more than one pair).
+        filtered_pairs = [item for index, item in enumerate(filtered_pairs) if index not in to_delete]
+
+        # remove pairs that are likely duplicates of others.
+        non_duplicated_pairs = [item for index, item in enumerate(filtered_pairs) if index not in to_delete]
path: src/extraction/features/extract.py
user: login: stijnvermeeren-swisstopo
body: Something is very wrong here. We filter out the `to_delete` indices three times, but each time from a different, smaller list. This cannot be on purpose...
created_at: 2025-04-25T12:55:48Z
author_association: COLLABORATOR
diff_hunk: @@ -91,26 +91,143 @@ def process_page(self) -> list[ExtractedBorehole]:
             pair for pair in material_descriptions_sidebar_pairs if pair.score_match >= 0
         ]
 
-        to_delete = []
-        for i, pair in enumerate(material_descriptions_sidebar_pairs):
-            if any(
-                pair.material_description_rect.intersects(other_pair.material_description_rect)
-                for other_pair in material_descriptions_sidebar_pairs[i + 1 :]
-            ):
-                to_delete.append(i)
-
+        # remoooove pairs that have any of their elements (sidebar, material description) intersecting with others.
path: src/extraction/features/extract.py
user: login: Copilot
body: There appears to be a typo in the comment ('remoooove'); please correct it to 'remove' for clarity.
```suggestion
        # remove pairs that have any of their elements (sidebar, material description) intersecting with others.
```
created_at: 2025-04-28T05:45:34Z
author_association: CONTRIBUTOR
diff_hunk: @@ -91,26 +91,143 @@ def process_page(self) -> list[ExtractedBorehole]:
             pair for pair in material_descriptions_sidebar_pairs if pair.score_match >= 0
         ]
 
-        to_delete = []
-        for i, pair in enumerate(material_descriptions_sidebar_pairs):
-            if any(
-                pair.material_description_rect.intersects(other_pair.material_description_rect)
-                for other_pair in material_descriptions_sidebar_pairs[i + 1 :]
-            ):
-                to_delete.append(i)
-
+        # remoooove pairs that have any of their elements (sidebar, material description) intersecting with others.
+        to_delete = self._find_intersecting_indices(material_descriptions_sidebar_pairs)
         filtered_pairs = [
             item for index, item in enumerate(material_descriptions_sidebar_pairs) if index not in to_delete
         ]
 
+        # remove pairs that have all depths equal to None (only if there is more than one pair).
+        to_delete = self._find_no_depths_indices(filtered_pairs)
+        filtered_pairs = [item for index, item in enumerate(filtered_pairs) if index not in to_delete]
+
+        # remove pairs that are likely duplicates of others.
+        to_delete = self._find_duplicated_pairs_indices(filtered_pairs)
+        non_duplicated_pairs = [item for index, item in enumerate(filtered_pairs) if index not in to_delete]
+
         # We order the boreholes with the highest score first. When one borehole is actually present in the ground
         # truth, but more than one are detected, we want the most correct to be assigned
-        # TODO we actually should try to merge similar extracted boreholes to keep the most information possible.
         return [
             self._create_borehole_from_pair(pair)
-            for pair in sorted(filtered_pairs, key=lambda pair: pair.score_match, reverse=True)
+            for pair in sorted(non_duplicated_pairs, key=lambda pair: pair.score_match, reverse=True)
+        ]
+
+    def _find_intersecting_indices(
+        self, material_descriptions_sidebar_pairs: list[MaterialDescriptionRectWithSidebar]
+    ) -> list[int]:
+        """Identifies overlapping material descriptions or sidebars.
+
+        This function scans through all material description/sidebar pairs and returns a list of indices
+        that should be removed due to overlaps. If an intersection is found between two elements, only
+        the first (lower-indexed) element is marked for deletion.
+
+        Args:
+            material_descriptions_sidebar_pairs (list[MaterialDescriptionRectWithSidebar]): a list of pairs consisting
+                of a material description rectangle and an optional sidebar.
+
+        Returns:
+            list[int]: The indices of elements that overlap with others and should be removed.
+        """
+        to_delete = []
+
+        def intersects_any(rect, others):
+            return any(rect.intersects(other) for other in others if other is not None)
+
+        for i, pair in enumerate(material_descriptions_sidebar_pairs):
+            mat_rect = pair.material_description_rect
+            sidebar_rect = pair.sidebar.rect() if pair.sidebar else None
+
+            # Build the list of other rectangles
+            remaining_pairs = material_descriptions_sidebar_pairs[i + 1 :]
+            other_mat_rects = [p.material_description_rect for p in remaining_pairs]
+            other_sidebar_rects = [p.sidebar.rect() if p.sidebar else None for p in remaining_pairs]
+
+            # Check all conditions
+            if (
+                intersects_any(mat_rect, other_mat_rects)
+                or intersects_any(mat_rect, other_sidebar_rects)
+                or (sidebar_rect and intersects_any(sidebar_rect, other_mat_rects))
+                or (sidebar_rect and intersects_any(sidebar_rect, other_sidebar_rects))
+            ):
+                to_delete.append(i)
+        return to_delete
+
+    def _find_no_depths_indices(self, filtered_pairs: list[MaterialDescriptionRectWithSidebar]) -> list[int]:
+        """Identify indices of pairs that have no depth intervals.
+
+        Pairs are deleted only if there is more than one element in filtered_pairs.
+
+        Args:
+            filtered_pairs (list[MaterialDescriptionRectWithSidebar]): List of material description/sidebar pairs.
+
+        Returns:
+            list[int]: Indices of pairs where all depth intervals are None.
+        """
+        # if only one pair is detected, we should keep it
+        if len(filtered_pairs) <= 1:
+            return []
+
+        all_interval_lists = [
+            [interval.depth_interval for interval in self._get_interval_block_pairs(pair)] for pair in filtered_pairs
+        ]
+
+        to_delete = [
+            i
+            for i, interval_list in enumerate(all_interval_lists)
+            if all(interval is None for interval in interval_list)
+        ]
+
+        return to_delete
+
+    def _find_duplicated_pairs_indices(self, filtered_pairs: list[MaterialDescriptionRectWithSidebar]) -> list[int]:
+        """Identify indices of pairs that are likely duplicates.
+
+        This is done because the informations about the same borehole could be represented in multiple ways.
+        We only want to keep multiple pairs if they belong to different boreholes in the given pdf page.
path: src/extraction/features/extract.py
user: login: Copilot
body: Consider revising 'informations' to 'information' as it is uncountable.
```suggestion
        This is done because the information about the same borehole could be represented in multiple ways.
```
created_at: 2025-04-28T05:45:34Z
author_association: CONTRIBUTOR
diff_hunk: @@ -91,26 +91,143 @@ def process_page(self) -> list[ExtractedBorehole]:
             pair for pair in material_descriptions_sidebar_pairs if pair.score_match >= 0
         ]
 
-        to_delete = []
-        for i, pair in enumerate(material_descriptions_sidebar_pairs):
-            if any(
-                pair.material_description_rect.intersects(other_pair.material_description_rect)
-                for other_pair in material_descriptions_sidebar_pairs[i + 1 :]
-            ):
-                to_delete.append(i)
-
+        # remove pairs that have any of their elements (sidebar, material description) intersecting with others.
+        to_delete = self._find_intersecting_indices(material_descriptions_sidebar_pairs)
         filtered_pairs = [
             item for index, item in enumerate(material_descriptions_sidebar_pairs) if index not in to_delete
         ]
 
+        # remoove pairs that have all depths equal to None (only if there is more than one pair).
+        to_delete = self._find_no_depths_indices(filtered_pairs)
path: src/extraction/features/extract.py
user: login: Copilot
body: Typo in comment: 'remoove' should be corrected to 'remove'.
```suggestion
        # remove pairs that have all depths equal to None (only if there is more than one pair).
```
created_at: 2025-04-28T05:49:02Z
author_association: CONTRIBUTOR
diff_hunk: @@ -91,26 +91,143 @@ def process_page(self) -> list[ExtractedBorehole]:
             pair for pair in material_descriptions_sidebar_pairs if pair.score_match >= 0
         ]
 
-        to_delete = []
-        for i, pair in enumerate(material_descriptions_sidebar_pairs):
-            if any(
-                pair.material_description_rect.intersects(other_pair.material_description_rect)
-                for other_pair in material_descriptions_sidebar_pairs[i + 1 :]
-            ):
-                to_delete.append(i)
-
+        # remove pairs that have any of their elements (sidebar, material description) intersecting with others.
+        to_delete = self._find_intersecting_indices(material_descriptions_sidebar_pairs)
         filtered_pairs = [
             item for index, item in enumerate(material_descriptions_sidebar_pairs) if index not in to_delete
         ]
 
+        # remoove pairs that have all depths equal to None (only if there is more than one pair).
+        to_delete = self._find_no_depths_indices(filtered_pairs)
path: src/extraction/features/extract.py
user: login: TicaGit
body: Here, it tries to correct a typo but actually deletes the line below too.
created_at: 2025-04-28T05:53:55Z
author_association: CONTRIBUTOR
diff_hunk: @@ -91,26 +91,143 @@ def process_page(self) -> list[ExtractedBorehole]:
             pair for pair in material_descriptions_sidebar_pairs if pair.score_match >= 0
         ]
 
-        to_delete = []
-        for i, pair in enumerate(material_descriptions_sidebar_pairs):
-            if any(
-                pair.material_description_rect.intersects(other_pair.material_description_rect)
-                for other_pair in material_descriptions_sidebar_pairs[i + 1 :]
-            ):
-                to_delete.append(i)
-
+        # remove pairs that have any of their elements (sidebar, material description) intersecting with others.
+        to_delete = self._find_intersecting_indices(material_descriptions_sidebar_pairs)
         filtered_pairs = [
             item for index, item in enumerate(material_descriptions_sidebar_pairs) if index not in to_delete
         ]
 
+        # remove pairs that have all depths equal to None (only if there is more than one pair).
+        to_delete = self._find_no_depths_indices(filtered_pairs)
+        filtered_pairs = [item for index, item in enumerate(filtered_pairs) if index not in to_delete]
+
+        # remove pairs that are likely duplicates of others.
+        to_delete = self._find_duplicated_pairs_indices(filtered_pairs)
+        non_duplicated_pairs = [item for index, item in enumerate(filtered_pairs) if index not in to_delete]
+
         # We order the boreholes with the highest score first. When one borehole is actually present in the ground
         # truth, but more than one are detected, we want the most correct to be assigned
-        # TODO we actually should try to merge similar extracted boreholes to keep the most information possible.
         return [
             self._create_borehole_from_pair(pair)
-            for pair in sorted(filtered_pairs, key=lambda pair: pair.score_match, reverse=True)
+            for pair in sorted(non_duplicated_pairs, key=lambda pair: pair.score_match, reverse=True)
+        ]
+
+    def _find_intersecting_indices(
+        self, material_descriptions_sidebar_pairs: list[MaterialDescriptionRectWithSidebar]
+    ) -> list[int]:
+        """Identifies overlapping material descriptions or sidebars.
+
+        This function scans through all material description/sidebar pairs and returns a list of indices
+        that should be removed due to overlaps. If an intersection is found between two elements, only
+        the first (lower-indexed) element is marked for deletion.
+
+        Args:
+            material_descriptions_sidebar_pairs (list[MaterialDescriptionRectWithSidebar]): a list of pairs consisting
+                of a material description rectangle and an optional sidebar.
+
+        Returns:
+            list[int]: The indices of elements that overlap with others and should be removed.
+        """
+        to_delete = []
+
+        def intersects_any(rect, others):
+            return any(rect.intersects(other) for other in others if other is not None)
+
+        for i, pair in enumerate(material_descriptions_sidebar_pairs):
+            mat_rect = pair.material_description_rect
+            sidebar_rect = pair.sidebar.rect() if pair.sidebar else None
+
+            # Build the list of other rectangles
+            remaining_pairs = material_descriptions_sidebar_pairs[i + 1 :]
+            other_mat_rects = [p.material_description_rect for p in remaining_pairs]
+            other_sidebar_rects = [p.sidebar.rect() if p.sidebar else None for p in remaining_pairs]
+
+            # Check all conditions
+            if (
+                intersects_any(mat_rect, other_mat_rects)
+                or intersects_any(mat_rect, other_sidebar_rects)
+                or (sidebar_rect and intersects_any(sidebar_rect, other_mat_rects))
+                or (sidebar_rect and intersects_any(sidebar_rect, other_sidebar_rects))
+            ):
+                to_delete.append(i)
+        return to_delete
+
+    def _find_no_depths_indices(self, filtered_pairs: list[MaterialDescriptionRectWithSidebar]) -> list[int]:
+        """Identify indices of pairs that have no depth intervals.
+
+        Pairs are deleted only if there is more than one element in filtered_pairs.
+
+        Args:
+            filtered_pairs (list[MaterialDescriptionRectWithSidebar]): List of material description/sidebar pairs.
+
+        Returns:
+            list[int]: Indices of pairs where all depth intervals are None.
+        """
+        # if only one pair is detected, we should keep it
+        if len(filtered_pairs) <= 1:
+            return []
+
+        all_interval_lists = [
+            [interval.depth_interval for interval in self._get_interval_block_pairs(pair)] for pair in filtered_pairs
+        ]
+
+        to_delete = [
+            i
+            for i, interval_list in enumerate(all_interval_lists)
+            if all(interval is None for interval in interval_list)
+        ]
+
+        return to_delete
+
+    def _find_duplicated_pairs_indices(self, filtered_pairs: list[MaterialDescriptionRectWithSidebar]) -> list[int]:
+        """Identify indices of pairs that are likely duplicates.
+
+        This is done because the informations about the same borehole could be represented in multiple ways.
+        We only want to keep multiple pairs if they belong to different boreholes in the given pdf page.
+        If duplication is found, we delete the first element, as the pairs are sorted from lowest score first.
+
+        Args:
+            filtered_pairs (list[MaterialDescriptionRectWithSidebar]): The filtered pairs.
+
+        Returns:
+            list[int]: A list containing the indexes of pairs that are duplicated and that should be deleted.
+        """
+        if len(filtered_pairs) <= 1:
+            # only one pair, can't be a duplicate
+            return []
+
+        all_interval_lists = [
+            [interval.depth_interval for interval in self._get_interval_block_pairs(pair)] for pair in filtered_pairs
         ]
 
+        # create sets with all the depths appearing in the interval list
+        all_depths_set = []
path: src/extraction/features/extract.py
user: login: stijnvermeeren-swisstopo
body: The variable name suggests that this is a set, whereas it is actually a list of sets. Maybe something like `all_depth_sets` or `list_of_depth_sets` would be clearer?
created_at: 2025-04-28T09:14:05Z
author_association: COLLABORATOR
diff_hunk: @@ -91,26 +91,143 @@ def process_page(self) -> list[ExtractedBorehole]:
             pair for pair in material_descriptions_sidebar_pairs if pair.score_match >= 0
         ]
 
-        to_delete = []
-        for i, pair in enumerate(material_descriptions_sidebar_pairs):
-            if any(
-                pair.material_description_rect.intersects(other_pair.material_description_rect)
-                for other_pair in material_descriptions_sidebar_pairs[i + 1 :]
-            ):
-                to_delete.append(i)
-
+        # remove pairs that have any of their elements (sidebar, material description) intersecting with others.
+        to_delete = self._find_intersecting_indices(material_descriptions_sidebar_pairs)
         filtered_pairs = [
             item for index, item in enumerate(material_descriptions_sidebar_pairs) if index not in to_delete
         ]
 
+        # remove pairs that have all depths equal to None (only if there is more than one pair).
+        to_delete = self._find_no_depths_indices(filtered_pairs)
+        filtered_pairs = [item for index, item in enumerate(filtered_pairs) if index not in to_delete]
+
+        # remove pairs that are likely duplicates of others.
+        to_delete = self._find_duplicated_pairs_indices(filtered_pairs)
+        non_duplicated_pairs = [item for index, item in enumerate(filtered_pairs) if index not in to_delete]
+
         # We order the boreholes with the highest score first. When one borehole is actually present in the ground
         # truth, but more than one are detected, we want the most correct to be assigned
-        # TODO we actually should try to merge similar extracted boreholes to keep the most information possible.
         return [
             self._create_borehole_from_pair(pair)
-            for pair in sorted(filtered_pairs, key=lambda pair: pair.score_match, reverse=True)
+            for pair in sorted(non_duplicated_pairs, key=lambda pair: pair.score_match, reverse=True)
+        ]
+
+    def _find_intersecting_indices(
+        self, material_descriptions_sidebar_pairs: list[MaterialDescriptionRectWithSidebar]
+    ) -> list[int]:
+        """Identifies overlapping material descriptions or sidebars.
+
+        This function scans through all material description/sidebar pairs and returns a list of indices
+        that should be removed due to overlaps. If an intersection is found between two elements, only
+        the first (lower-indexed) element is marked for deletion.
+
+        Args:
+            material_descriptions_sidebar_pairs (list[MaterialDescriptionRectWithSidebar]): a list of pairs consisting
+                of a material description rectangle and an optional sidebar.
+
+        Returns:
+            list[int]: The indices of elements that overlap with others and should be removed.
+        """
+        to_delete = []
+
+        def intersects_any(rect, others):
+            return any(rect.intersects(other) for other in others if other is not None)
+
+        for i, pair in enumerate(material_descriptions_sidebar_pairs):
+            mat_rect = pair.material_description_rect
+            sidebar_rect = pair.sidebar.rect() if pair.sidebar else None
+
+            # Build the list of other rectangles
+            remaining_pairs = material_descriptions_sidebar_pairs[i + 1 :]
+            other_mat_rects = [p.material_description_rect for p in remaining_pairs]
+            other_sidebar_rects = [p.sidebar.rect() if p.sidebar else None for p in remaining_pairs]
+
+            # Check all conditions
+            if (
+                intersects_any(mat_rect, other_mat_rects)
+                or intersects_any(mat_rect, other_sidebar_rects)
+                or (sidebar_rect and intersects_any(sidebar_rect, other_mat_rects))
+                or (sidebar_rect and intersects_any(sidebar_rect, other_sidebar_rects))
+            ):
+                to_delete.append(i)
+        return to_delete
+
+    def _find_no_depths_indices(self, filtered_pairs: list[MaterialDescriptionRectWithSidebar]) -> list[int]:
+        """Identify indices of pairs that have no depth intervals.
+
+        Pairs are deleted only if there is more than one element in filtered_pairs.
+
+        Args:
+            filtered_pairs (list[MaterialDescriptionRectWithSidebar]): List of material description/sidebar pairs.
+
+        Returns:
+            list[int]: Indices of pairs where all depth intervals are None.
+        """
+        # if only one pair is detected, we should keep it
+        if len(filtered_pairs) <= 1:
+            return []
+
+        all_interval_lists = [
+            [interval.depth_interval for interval in self._get_interval_block_pairs(pair)] for pair in filtered_pairs
path: src/extraction/features/extract.py
user: login: stijnvermeeren-swisstopo
body: This filters out both pairs that don't have any sidebar, as well as "indicator" sidebars. In some cases however, I think it is worthwhile to keep the indicator sidebars. One example is 1488.pdf, which you list as a "false positive removed", but I would actually argue that this change is not so good, because even thought the F1 score improves, we actually lose some valuable descriptions at the top of the borehole profile.

If I change/simplify the condition to 
```
        to_delete = [
            i
            for i, pair in enumerate(filtered_pairs)
            if not pair.sidebar
        ]
```
then we keep these descriptions. However, it also reverts the improvements for e.g. 14273.pdf. Maybe if we do the duplicate check not just for depth values, but also for the indicator values, then we might get the best of both approaches? Could you try that out?
created_at: 2025-04-28T09:37:23Z
author_association: COLLABORATOR
diff_hunk: @@ -91,26 +91,143 @@ def process_page(self) -> list[ExtractedBorehole]:
             pair for pair in material_descriptions_sidebar_pairs if pair.score_match >= 0
         ]
 
-        to_delete = []
-        for i, pair in enumerate(material_descriptions_sidebar_pairs):
-            if any(
-                pair.material_description_rect.intersects(other_pair.material_description_rect)
-                for other_pair in material_descriptions_sidebar_pairs[i + 1 :]
-            ):
-                to_delete.append(i)
-
+        # remove pairs that have any of their elements (sidebar, material description) intersecting with others.
+        to_delete = self._find_intersecting_indices(material_descriptions_sidebar_pairs)
         filtered_pairs = [
             item for index, item in enumerate(material_descriptions_sidebar_pairs) if index not in to_delete
         ]
 
+        # remove pairs that have all depths equal to None (only if there is more than one pair).
+        to_delete = self._find_no_depths_indices(filtered_pairs)
+        filtered_pairs = [item for index, item in enumerate(filtered_pairs) if index not in to_delete]
+
+        # remove pairs that are likely duplicates of others.
+        to_delete = self._find_duplicated_pairs_indices(filtered_pairs)
+        non_duplicated_pairs = [item for index, item in enumerate(filtered_pairs) if index not in to_delete]
+
         # We order the boreholes with the highest score first. When one borehole is actually present in the ground
         # truth, but more than one are detected, we want the most correct to be assigned
-        # TODO we actually should try to merge similar extracted boreholes to keep the most information possible.
         return [
             self._create_borehole_from_pair(pair)
-            for pair in sorted(filtered_pairs, key=lambda pair: pair.score_match, reverse=True)
+            for pair in sorted(non_duplicated_pairs, key=lambda pair: pair.score_match, reverse=True)
+        ]
+
+    def _find_intersecting_indices(
+        self, material_descriptions_sidebar_pairs: list[MaterialDescriptionRectWithSidebar]
+    ) -> list[int]:
+        """Identifies overlapping material descriptions or sidebars.
+
+        This function scans through all material description/sidebar pairs and returns a list of indices
+        that should be removed due to overlaps. If an intersection is found between two elements, only
+        the first (lower-indexed) element is marked for deletion.
+
+        Args:
+            material_descriptions_sidebar_pairs (list[MaterialDescriptionRectWithSidebar]): a list of pairs consisting
+                of a material description rectangle and an optional sidebar.
+
+        Returns:
+            list[int]: The indices of elements that overlap with others and should be removed.
+        """
+        to_delete = []
+
+        def intersects_any(rect, others):
+            return any(rect.intersects(other) for other in others if other is not None)
+
+        for i, pair in enumerate(material_descriptions_sidebar_pairs):
+            mat_rect = pair.material_description_rect
+            sidebar_rect = pair.sidebar.rect() if pair.sidebar else None
+
+            # Build the list of other rectangles
+            remaining_pairs = material_descriptions_sidebar_pairs[i + 1 :]
+            other_mat_rects = [p.material_description_rect for p in remaining_pairs]
+            other_sidebar_rects = [p.sidebar.rect() if p.sidebar else None for p in remaining_pairs]
+
+            # Check all conditions
+            if (
+                intersects_any(mat_rect, other_mat_rects)
+                or intersects_any(mat_rect, other_sidebar_rects)
+                or (sidebar_rect and intersects_any(sidebar_rect, other_mat_rects))
+                or (sidebar_rect and intersects_any(sidebar_rect, other_sidebar_rects))
+            ):
+                to_delete.append(i)
+        return to_delete
+
+    def _find_no_depths_indices(self, filtered_pairs: list[MaterialDescriptionRectWithSidebar]) -> list[int]:
+        """Identify indices of pairs that have no depth intervals.
+
+        Pairs are deleted only if there is more than one element in filtered_pairs.
+
+        Args:
+            filtered_pairs (list[MaterialDescriptionRectWithSidebar]): List of material description/sidebar pairs.
+
+        Returns:
+            list[int]: Indices of pairs where all depth intervals are None.
+        """
+        # if only one pair is detected, we should keep it
+        if len(filtered_pairs) <= 1:
+            return []
+
+        all_interval_lists = [
+            [interval.depth_interval for interval in self._get_interval_block_pairs(pair)] for pair in filtered_pairs
path: src/extraction/features/extract.py
user: login: TicaGit
body: The reason I did not use your code snipet is because we can still identify depths in the descriptions, otherwise the whole idea of having adding material_description_rect_without_sidebar would be useless (as we would add them and remove them directly in the filtering).

I created a branch where I implemented your idea: fix/issue-130/sidebar-material-overlap-alternative. It works for the files that do contain LayerIdentifierSidebar but not for the other. So the files A1304.pdf, A11406.pdf, 1488.pdf and695265010-bp.pdf (zurich) are now NOT filtered (both boreholes remain). I would argue that this is worse, I think we should deleted those 2nd parasite boreholes, and that the problem with 1488.pdf is more related to the extraction process (it should not be detected as 2 boreholes in the first place). We can discuss this in a call if you want.
created_at: 2025-04-28T16:09:47Z
author_association: CONTRIBUTOR

PR comments:
user: login: github-actions[bot]
created_at: 2025-04-25T09:08:30Z
body: <!-- Pytest Coverage Comment: pytest -->
| Tests | Skipped | Failures | Errors | Time |
| ----- | ------- | -------- | -------- | ------------------ |
| 127 | 0 :zzz: | 0 :x: | 0 :fire: | 9.528s :stopwatch: |

author_association: NONE
user: login: TicaGit
created_at: 2025-04-30T07:08:03Z
body: Now neither of the two boreholes in 1488.pdf are deleted, but all of the other mentioned files are filtered I believe.
author_association: CONTRIBUTOR