Repository: climatepolicyradar/knowledge-graph
Pull request #356: Feature/pla 528 update pipeline to remove value error raised on non english
Author: login: THOR300
Created: 2025-04-15T10:27:26Z | Merged: 2025-04-16T09:35:08Z

This Pull Request: 
---
- Updates the inference pipeline to no longer run on non-english documents when the run configuration that's provided is all/some classifier specs. 
- This is as we are raising a ValueError when inference is attempted on non-english documents and thus in the prefect console the runs look failed which is confusing. 
- An example of this can be seen [here](https://app.prefect.cloud/account/4b1558a0-3c61-4849-8b18-3e97e0516d78/workspace/1753b4f0-6221-4f6a-9233-b146518b4545/runs/flow-run/5f8502ac-6ecc-4979-8650-b4d60c749425). 
- Testing on staging: [run](https://app.prefect.cloud/account/4b1558a0-3c61-4849-8b18-3e97e0516d78/workspace/1753b4f0-6221-4f6a-9233-b146518b4545/runs/flow-run/b80e96f6-a582-4c25-a2bc-236cfb0234ef?entity_id=b80e96f6-a582-4c25-a2bc-236cfb0234ef)


Diff:
diff --git a/flows/inference.py b/flows/inference.py
index b562f4363..847771ed0 100644
--- a/flows/inference.py
+++ b/flows/inference.py
@@ -24,7 +24,11 @@
 from pydantic import SecretStr
 from wandb.sdk.wandb_run import Run
 
-from flows.utils import SlackNotify, get_file_stems_for_document_id
+from flows.utils import (
+    SlackNotify,
+    filter_non_english_language_file_stems,
+    get_file_stems_for_document_id,
+)
 from scripts.cloud import (
     AwsEnv,
     ClassifierSpec,
@@ -198,7 +202,10 @@ def determine_file_stems(
     elif use_new_and_updated:
         requested_document_ids = get_latest_ingest_documents(config)
     elif requested_document_ids is None:
-        return current_bucket_file_stems
+        current_bucket_file_stems__filtered = filter_non_english_language_file_stems(
+            file_stems=current_bucket_file_stems
+        )
+        return current_bucket_file_stems__filtered
 
     assert config.cache_bucket
 
@@ -412,11 +419,26 @@ async def run_classifier_inference_on_document(
     document = load_document(config, file_stem)
     print(f"Loaded document with file stem {file_stem}")
 
-    # FIXME: This makes the logs look like the run has failed and can happen like 300
-    # times per batch.
+    # Handle documents with no text and no language
+    if (
+        not document.languages
+        and document.pdf_data is None
+        and document.html_data is None
+    ):
+        store_labels(
+            config=config,
+            labels=[],
+            file_stem=file_stem,
+            classifier_name=classifier_name,
+            classifier_alias=classifier_alias,
+        )
+
+        return None
+
+    # Raise on non-English documents
     if document.languages != ["en"]:
         raise ValueError(
-            f"Cannot run inference on {file_stem} as it has non-english language: "
+            f"Cannot run inference on {file_stem} as it has non-English language: "
             f"{document.languages}"
         )
 
diff --git a/flows/utils.py b/flows/utils.py
index da838ddc5..e66477c1d 100644
--- a/flows/utils.py
+++ b/flows/utils.py
@@ -16,6 +16,8 @@
 DocumentImportId: TypeAlias = str
 DocumentStem: TypeAlias = str
 
+DOCUMENT_ID_PATTERN = re.compile(r"^((?:[^.]+\.){3}[^._]+)")
+
 
 def file_name_from_path(path: str) -> str:
     """Get the file name from a path without the path or extension"""
@@ -235,3 +237,34 @@ def _s3_object_write_bytes(s3_uri: str, bytes: BytesIO) -> None:
     _ = s3.put_object(
         Bucket=bucket, Key=key, Body=bytes, ContentType="application/json"
     )
+
+
+def is_file_stem_for_english_language_document(
+    file_stem: DocumentStem,
+    file_stems: list[DocumentStem],
+    english_translation_suffix: str = "_translated_en",
+) -> bool:
+    """
+    Check if a file stem is in English language.
+
+    - If the file stem has the translated_en suffix then we can infer that it's english.
+    - If there's a translated version of the document in the list, we can infer that it's
+        not english.
+    """
+    if english_translation_suffix in file_stem:
+        return True
+    if file_stem + english_translation_suffix in file_stems:
+        return False
+    return True
+
+
+def filter_non_english_language_file_stems(
+    file_stems: list[DocumentStem],
+) -> list[DocumentStem]:
+    """Filter out file stems that are for non-English language documents."""
+    return list(
+        filter(
+            lambda f: is_file_stem_for_english_language_document(f, file_stems),
+            file_stems,
+        )
+    )
diff --git a/tests/flows/test_inference.py b/tests/flows/test_inference.py
index 86ec8b20f..29adbb798 100644
--- a/tests/flows/test_inference.py
+++ b/tests/flows/test_inference.py
@@ -47,8 +47,16 @@ def test_list_bucket_file_stems(test_config, mock_bucket_documents):
 @pytest.mark.parametrize(
     ("doc_ids", "bucket_ids", "expected"),
     [
-        (["1"], ["1", "2", "3"], ["1"]),
-        (None, ["1"], ["1"]),
+        (
+            ["AF.document.002MMUCR.n0000"],
+            [
+                "AF.document.002MMUCR.n0000",
+                "AF.document.AFRDG00038.n0000",
+                "CCLW.document.i00001313.n0000",
+            ],
+            ["AF.document.002MMUCR.n0000"],
+        ),
+        (None, ["AF.document.002MMUCR.n0000"], ["AF.document.002MMUCR.n0000"]),
     ],
 )
 def test_determine_file_stems(test_config, doc_ids, bucket_ids, expected):
@@ -66,8 +74,14 @@ def test_determine_file_stems__error(test_config):
         determine_file_stems(
             config=test_config,
             use_new_and_updated=False,
-            requested_document_ids=["1", "2"],
-            current_bucket_file_stems=["3", "4"],
+            requested_document_ids=[
+                "AF.document.002MMUCR.n0000",
+                "AF.document.AFRDG00038.n00002",
+            ],
+            current_bucket_file_stems=[
+                "CCLW.document.i00001313.n0000",
+                "AF.document.002MMUCR.n0000",
+            ],
         )
 
 
diff --git a/tests/flows/test_utils.py b/tests/flows/test_utils.py
index 36969a63f..d049a3dc3 100644
--- a/tests/flows/test_utils.py
+++ b/tests/flows/test_utils.py
@@ -1,5 +1,6 @@
 import os
 import re
+import time
 from io import BytesIO
 from pathlib import Path
 
@@ -9,6 +10,7 @@
 from flows.utils import (
     SlackNotify,
     file_name_from_path,
+    filter_non_english_language_file_stems,
     get_file_stems_for_document_id,
     get_labelled_passage_paths,
     iterate_batch,
@@ -167,3 +169,56 @@ def test_get_labelled_passage_paths(test_config, mock_s3_client, mock_bucket) ->
             f"s3://{test_config.cache_bucket}/{test_config.document_target_prefix}/{classifier_spec.name}/{classifier_spec.alias}/CCLW.executive.10083.rtl_190.json",
         ]
     )
+
+
+def test_filter_non_english_file_stems() -> None:
+    """Test that we can successfully filter out non-English file stems."""
+
+    # Test we can filter out non-English file stems
+    file_stems = [
+        "AF.document.002MMUCR.n0000",
+        "AF.document.AFRDG00038.n0000_translated_en",
+        "AF.document.AFRDG00038.n0000",
+        "CCLW.document.i00001313.n0000",
+        "CCLW.executive.10491.5394",
+        "CCLW.executive.rtl_98.rtl_338",
+        "CCLW.executive.rtl_98.rtl_338_translated_en",
+        "CCLW.legislative.1046.0",
+        "CPR.document.i00000918.n0000",
+        "GCF.document.FP018_13290.6302",
+        "GEF.document.2480.n0000_translated_en",
+        "GEF.document.2480.n0000",
+        "OEP.document.i00000157.n0000",
+        "UNFCCC.document.i00000546.n0000",
+        "UNFCCC.non-party.1243.0",
+    ]
+
+    filtered_file_stems = filter_non_english_language_file_stems(file_stems=file_stems)
+
+    assert filtered_file_stems == [
+        "AF.document.002MMUCR.n0000",
+        "AF.document.AFRDG00038.n0000_translated_en",
+        "CCLW.document.i00001313.n0000",
+        "CCLW.executive.10491.5394",
+        "CCLW.executive.rtl_98.rtl_338_translated_en",
+        "CCLW.legislative.1046.0",
+        "CPR.document.i00000918.n0000",
+        "GCF.document.FP018_13290.6302",
+        "GEF.document.2480.n0000_translated_en",
+        "OEP.document.i00000157.n0000",
+        "UNFCCC.document.i00000546.n0000",
+        "UNFCCC.non-party.1243.0",
+    ]
+
+    # Test that we can filter quickly even on very long lists
+    file_stems = []
+    for i in list(range(7_000)):
+        file_stems.append(f"AF.document.{i}.n0000")
+        if i % 2 == 0:
+            file_stems.append(f"AF.document.{i}.n0000_translated_en")
+
+    start_time = time.time()
+    _ = filter_non_english_language_file_stems(file_stems=file_stems)
+    end_time = time.time()
+
+    assert end_time - start_time < 1, "Filtering took too long"


Reviews:
user: login: mpjuhasz
body: 🚀 
state: APPROVED
author_association: CONTRIBUTOR
submitted_at: 2025-04-15T12:30:05Z
user: login: THOR300
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-04-15T12:34:21Z
user: login: jesse-c
body: Looks good so far!
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-04-15T16:21:10Z
user: login: THOR300
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-04-16T07:36:20Z
user: login: mpjuhasz
body: Looks good 🕺 
state: APPROVED
author_association: CONTRIBUTOR
submitted_at: 2025-04-16T08:03:20Z
user: login: THOR300
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-04-16T08:08:05Z
user: login: jesse-c
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-04-16T09:18:26Z
user: login: jesse-c
body: LGTM

Filtering over removing the exception is a good approach.
state: APPROVED
author_association: CONTRIBUTOR
submitted_at: 2025-04-16T09:21:04Z

Review comments:
diff_hunk: @@ -16,6 +17,8 @@
 DocumentImportId: TypeAlias = str
 DocumentStem: TypeAlias = str
 
+DOCUMENT_ID_PATTERN = re.compile(r"^((?:[^.]+\.){3}[^._]+)")
path: flows/utils.py
user: login: mpjuhasz
body: From memory, there can be underscores in the document ids -- would this not fail to match those? 
created_at: 2025-04-15T12:19:58Z
author_association: CONTRIBUTOR
diff_hunk: @@ -235,3 +238,34 @@ def _s3_object_write_bytes(s3_uri: str, bytes: BytesIO) -> None:
     _ = s3.put_object(
         Bucket=bucket, Key=key, Body=bytes, ContentType="application/json"
     )
+
+
+def filter_non_english_language_file_stems(
+    file_stems: list[DocumentStem],
+) -> list[DocumentStem]:
+    """Filter non-english related file stems."""
+
+    grouped = defaultdict(list)
+
+    for f in file_stems:
+        match = DOCUMENT_ID_PATTERN.match(f)
+        if match:
+            document_id = match.group(1)
+            grouped[document_id].append(f)
+        else:
+            grouped["no_match"].append(f)
+
+    if grouped["no_match"]:
+        raise ValueError(
+            f"File stems that do not match the pattern: {grouped['no_match']}"
+        )
+
+    filtered_file_stems = []
+    for group in grouped.values():
+        translated_file_stems = [f for f in group if "_translated" in f]
+        if translated_file_stems:
+            filtered_file_stems.extend(translated_file_stems)
+        else:
+            filtered_file_stems.extend(group)
path: flows/utils.py
user: login: mpjuhasz
body: I remember reviewing this same logic elsewhere 🤔 is this duplicated, or am I remembering wrong?
created_at: 2025-04-15T12:29:22Z
author_association: CONTRIBUTOR
diff_hunk: @@ -16,6 +17,8 @@
 DocumentImportId: TypeAlias = str
 DocumentStem: TypeAlias = str
 
+DOCUMENT_ID_PATTERN = re.compile(r"^((?:[^.]+\.){3}[^._]+)")
path: flows/utils.py
user: login: mpjuhasz
body: Never mind me -- saw the tests, and wrapped my head around this, looks good 👌 
created_at: 2025-04-15T12:29:57Z
author_association: CONTRIBUTOR
diff_hunk: @@ -235,3 +238,34 @@ def _s3_object_write_bytes(s3_uri: str, bytes: BytesIO) -> None:
     _ = s3.put_object(
         Bucket=bucket, Key=key, Body=bytes, ContentType="application/json"
     )
+
+
+def filter_non_english_language_file_stems(
+    file_stems: list[DocumentStem],
+) -> list[DocumentStem]:
+    """Filter non-english related file stems."""
+
+    grouped = defaultdict(list)
+
+    for f in file_stems:
+        match = DOCUMENT_ID_PATTERN.match(f)
+        if match:
+            document_id = match.group(1)
+            grouped[document_id].append(f)
+        else:
+            grouped["no_match"].append(f)
+
+    if grouped["no_match"]:
+        raise ValueError(
+            f"File stems that do not match the pattern: {grouped['no_match']}"
+        )
+
+    filtered_file_stems = []
+    for group in grouped.values():
+        translated_file_stems = [f for f in group if "_translated" in f]
+        if translated_file_stems:
+            filtered_file_stems.extend(translated_file_stems)
+        else:
+            filtered_file_stems.extend(group)
path: flows/utils.py
user: login: THOR300
body: There is a `get_file_stems_for_document_id` function as well but that was where we had a list of document ids we want to process (so we're adding translated file stems). 

In this instance we have all the file stems for the bucket and want to filter out the file stems related to non-english documents.                    
created_at: 2025-04-15T12:34:21Z
author_association: CONTRIBUTOR
diff_hunk: @@ -412,8 +419,21 @@ async def run_classifier_inference_on_document(
     document = load_document(config, file_stem)
     print(f"Loaded document with file stem {file_stem}")
 
-    # FIXME: This makes the logs look like the run has failed and can happen like 300
-    # times per batch.
+    # Handle documents with no text or language
+    if (
+        not document.languages
+        and document.pdf_data is None
+        and document.html_data is None
+    ):
path: flows/inference.py
user: login: jesse-c
body: **fix:** The boolean logic doesn't match the comment. Something like this would:
```suggestion
    # Handle documents with no text or language
    if (
        (not document.languages) or
        (document.pdf_data is None
        and document.html_data is None)
    ):
```
created_at: 2025-04-15T15:53:38Z
author_association: CONTRIBUTOR
diff_hunk: @@ -235,3 +238,34 @@ def _s3_object_write_bytes(s3_uri: str, bytes: BytesIO) -> None:
     _ = s3.put_object(
         Bucket=bucket, Key=key, Body=bytes, ContentType="application/json"
     )
+
+
+def filter_non_english_language_file_stems(
+    file_stems: list[DocumentStem],
+) -> list[DocumentStem]:
+    """Filter non-english related file stems."""
path: flows/utils.py
user: login: jesse-c
body: **quibble:** Correct spelling. Similarly elsewhere too, in the PR.
```suggestion
    """Filter non-English related file stems."""
```
created_at: 2025-04-15T15:59:12Z
author_association: CONTRIBUTOR
diff_hunk: @@ -235,3 +238,34 @@ def _s3_object_write_bytes(s3_uri: str, bytes: BytesIO) -> None:
     _ = s3.put_object(
         Bucket=bucket, Key=key, Body=bytes, ContentType="application/json"
     )
+
+
+def filter_non_english_language_file_stems(
path: flows/utils.py
user: login: jesse-c
body: **suggestion:** A FP approach would be to have a function that returns a boolean if it's English or not, and then use Python's builtin `filter`.
created_at: 2025-04-15T16:00:22Z
author_association: CONTRIBUTOR
diff_hunk: @@ -235,3 +238,34 @@ def _s3_object_write_bytes(s3_uri: str, bytes: BytesIO) -> None:
     _ = s3.put_object(
         Bucket=bucket, Key=key, Body=bytes, ContentType="application/json"
     )
+
+
+def filter_non_english_language_file_stems(
+    file_stems: list[DocumentStem],
+) -> list[DocumentStem]:
+    """Filter non-english related file stems."""
+
+    grouped = defaultdict(list)
+
+    for f in file_stems:
+        match = DOCUMENT_ID_PATTERN.match(f)
+        if match:
+            document_id = match.group(1)
+            grouped[document_id].append(f)
+        else:
+            grouped["no_match"].append(f)
+
+    if grouped["no_match"]:
+        raise ValueError(
+            f"File stems that do not match the pattern: {grouped['no_match']}"
+        )
path: flows/utils.py
user: login: jesse-c
body: **polish:** This function does not just filtering, but validation as well. I'd move validation to a separate function, or update the function name to reflect its dual purposes.
created_at: 2025-04-15T16:06:12Z
author_association: CONTRIBUTOR
diff_hunk: @@ -235,3 +238,34 @@ def _s3_object_write_bytes(s3_uri: str, bytes: BytesIO) -> None:
     _ = s3.put_object(
         Bucket=bucket, Key=key, Body=bytes, ContentType="application/json"
     )
+
+
+def filter_non_english_language_file_stems(
+    file_stems: list[DocumentStem],
+) -> list[DocumentStem]:
+    """Filter non-english related file stems."""
+
+    grouped = defaultdict(list)
+
+    for f in file_stems:
+        match = DOCUMENT_ID_PATTERN.match(f)
+        if match:
+            document_id = match.group(1)
+            grouped[document_id].append(f)
+        else:
+            grouped["no_match"].append(f)
+
+    if grouped["no_match"]:
+        raise ValueError(
+            f"File stems that do not match the pattern: {grouped['no_match']}"
+        )
+
+    filtered_file_stems = []
+    for group in grouped.values():
+        translated_file_stems = [f for f in group if "_translated" in f]
+        if translated_file_stems:
+            filtered_file_stems.extend(translated_file_stems)
+        else:
+            filtered_file_stems.extend(group)
path: flows/utils.py
user: login: jesse-c
body: **suggestion:** I find the data structure choice of a list confusing, at least without it being recorded that it should only ever have length of `[1,2]`, where there's either just the original or the original + translated. I'd either use a different data structure, or add a comment.

Or, can it be length of > 2?
created_at: 2025-04-15T16:17:50Z
author_association: CONTRIBUTOR
diff_hunk: @@ -412,8 +419,21 @@ async def run_classifier_inference_on_document(
     document = load_document(config, file_stem)
     print(f"Loaded document with file stem {file_stem}")
 
-    # FIXME: This makes the logs look like the run has failed and can happen like 300
-    # times per batch.
+    # Handle documents with no text or language
+    if (
+        not document.languages
+        and document.pdf_data is None
+        and document.html_data is None
+    ):
path: flows/inference.py
user: login: THOR300
body: Ahh yes good post, it's actually the comment that's the wrong definition! 
created_at: 2025-04-16T07:36:20Z
author_association: CONTRIBUTOR
diff_hunk: @@ -235,3 +238,34 @@ def _s3_object_write_bytes(s3_uri: str, bytes: BytesIO) -> None:
     _ = s3.put_object(
         Bucket=bucket, Key=key, Body=bytes, ContentType="application/json"
     )
+
+
+def filter_non_english_language_file_stems(
path: flows/utils.py
user: login: THOR300
body: Updated to using a functional approach, let me know what you think! I like the steer, looking much better now. 
created_at: 2025-04-16T08:08:05Z
author_association: CONTRIBUTOR
diff_hunk: @@ -235,3 +238,34 @@ def _s3_object_write_bytes(s3_uri: str, bytes: BytesIO) -> None:
     _ = s3.put_object(
         Bucket=bucket, Key=key, Body=bytes, ContentType="application/json"
     )
+
+
+def filter_non_english_language_file_stems(
path: flows/utils.py
user: login: jesse-c
body: Agreed! It makes the data and control flows so direct. To continue the FP path, I'd remove `filter_non_english_language_file_stems` and move its code inline. Since `filter` itself doesn't need to be tested, that function doesn't need to be tested (or exist!).

Sadly vanilla Python means you need to have the `list()` wrapper, so it can look verbose.

The Clean Code approach is to make functions for everything, but the FP is to use as many built-ins as possible, and have a direct flow.

Feel free to ignore the rest of this comment, but here's some expanded things about this. An example that has a pipe operator that's in most major FP languages:

```
stems
|> map(parse_str_to_stem_id)
|> filter(is_file_stem_for_english_language_document)
|> batch(size=10)
|> task(partial_update_docs)
```

Where `str_to_stem_id` is the previous validation about if it's a `DOCUMENT_ID` or not, and also the benefit of parsing[^1]. The rest are also vague allusions to what we're doing. It's honestly so nice having all the batches, tasks, streaming, etc as nice built-in ways of doing stuff, and takes care of all the manual stuff we've written about gathering tasks and the like.

A tidbit with this is that this is also the Prefect transaction (with optional rollbacks) way of viewing it. https://fsharpforfunandprofit.com/rop/ is a fun way to view it.

[^1]: https://lexi-lambda.github.io/blog/2019/11/05/parse-don-t-validate/
created_at: 2025-04-16T09:18:26Z
author_association: CONTRIBUTOR

PR comments:
user: login: linear[bot]
created_at: 2025-04-15T10:27:28Z
body: <p><a href="https://linear.app/climate-policy-radar/issue/PLA-528/update-pipeline-to-remove-value-error-raised-on-non-english-documents">PLA-528 Update pipeline to remove Value Error raised on non-english documents</a></p>
author_association: NONE
user: login: THOR300
created_at: 2025-04-15T14:23:49Z
body: Tested on staging, unfortunately we have some documents that are still being presented from inference, this is as they have no related `${doc_id}_translated_en` file as they have no source url and thus are not presented for translated: 
![image](https://github.com/user-attachments/assets/9ce84c86-6a0f-4660-92ad-f008395915ae)

Parser Code: 
- https://github.com/climatepolicyradar/navigator-document-parser/blob/main/cli/translate_outputs.py#L24

To solve this problem I've updated the conditional upon which we throw a validation error to just create empty outputs in this scenario. 
author_association: CONTRIBUTOR
user: login: jesse-c
created_at: 2025-04-15T15:07:04Z
body: Did not see that you were using staging! I just deployed my branch, sorry. Feel free to re-deploy your branch. LMK once you're done with staging 🙇🏻 
author_association: CONTRIBUTOR
user: login: THOR300
created_at: 2025-04-16T08:25:24Z
body: New prefect run on staging passing now without validation errors, I've checked in staging and all the outputs are for todays data so inference is running as expected 👍 : 
- https://app.prefect.cloud/account/4b1558a0-3c61-4849-8b18-3e97e0516d78/workspace/1753b4f0-6221-4f6a-9233-b146518b4545/runs/flow-run/1d74b622-20e9-4fef-84a9-ef5e1780d98c?entity_id=1d74b622-20e9-4fef-84a9-ef5e1780d98c
author_association: CONTRIBUTOR