Repository: openzim/python-scraperlib
Pull request #261: Add utility to deduplicate ZIM items and replace them with redirects at ZIM creation time
Author: login: benoit74
Created: 2025-05-06T07:12:45Z | Merged: 2025-05-09T11:52:21Z

Fix #33 

Note that this is kinda a resurrection of https://github.com/openzim/python-scraperlib/pull/86 where important things have already been discussed (e.g. the fact that we do not want to extend the Creator API but add a new distinct API to better trace memory issues)

Linked issue #33: Automatically redirect to articles with same checksum
As discussed in https://github.com/openzim/sotoki/pull/162#issuecomment-660452579, it actually seems a bit odd to handle duplicate files in the scrapers. We can instead have a system to redirect have a single copy of a resource and create redirects if that's being duplicated (or fail intelligently so we can handle). 
Linked issue comments: user: login: kelson42
created_at: 2020-07-18T18:26:58Z
body: To me, for the moment, such a feature should better be in python-scraperlib (or any higher level library) because:
* This is too smart to be done in the libzim
* I believe basically the scraper (not the libzim) should be able to do things in a clean manner
* I understand under certain special conditions the high level scraper might better rely for a certain range of articles of a lower level smart feature like this
author_association: CONTRIBUTOR
user: login: rgaudin
created_at: 2020-07-20T10:46:26Z
body: Also, as discussed with @kelson42, articles have no checksum in the ZIM. I was led to think that based on zimcheck's duplicates output but it's zimcheck calculating those.

What we could do is have a helper in scraperlib that calculates checksums, stores them and compares them to adjust behavior (create redirects?).
That would be extra and should be enabled on a subset of articles via some filtering pattern.
The main use case would be for zimit where the scraper has no control over the content. In this case, if the zimcheck reports duplicates, we could enable this mechanism in the recipe by specifying the filtering patterns.

This *feature* could have a HUGE impact on resources (CPU, RAM, potentially IO) so it's goal will be to clear duplicates for the case it **cannot** be done in the scraper. Non-generic scrapers should take care of duplicates themselves.
author_association: MEMBER
user: login: stale[bot]
created_at: 2020-10-10T03:01:58Z
body: This issue has been automatically marked as stale because it has not had recent activity. It will be now be reviewed manually. Thank you for your contributions.

author_association: NONE
user: login: benoit74
created_at: 2022-04-20T19:26:00Z
body: I will start to work on an implementation of this issue. Will open a PR once I have something ready to review. I will try to follow advises mentioned above
author_association: COLLABORATOR
user: login: stale[bot]
created_at: 2022-08-13T10:47:37Z
body: This issue has been automatically marked as stale because it has not had recent activity. It will be now be reviewed manually. Thank you for your contributions.

author_association: NONE
user: login: kelson42
created_at: 2023-12-16T17:30:34Z
body: Maybe we should better use aliases?
author_association: CONTRIBUTOR
user: login: rgaudin
created_at: 2023-12-16T17:37:20Z
body: > Maybe we should better use aliases?

Doesn't solve anything. We still don't know ahead of adding the entry that it's a _duplicate_ otherwise we'd probably do thing differently depending on the scraper: not include the resource, use an alias or a redirect.

author_association: MEMBER

Diff:
diff --git a/CHANGELOG.md b/CHANGELOG.md
index 1d182f8..1af4791 100644
--- a/CHANGELOG.md
+++ b/CHANGELOG.md
@@ -7,6 +7,10 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
 
 ## [Unreleased]
 
+### Added
+
+- New `zim.dedup.Deduplicator` class to handle automatic deduplication of content before adding to the ZIM (#33)
+
 ### Changed
 
 - Upgrade to wombat 3.8.11 (#256)
diff --git a/pyproject.toml b/pyproject.toml
index 85055e1..19c6a74 100644
--- a/pyproject.toml
+++ b/pyproject.toml
@@ -32,7 +32,9 @@ dependencies = [
   "pillow>=7.0.0,<12.0",
   "urllib3>=1.26.5,<2.4.0",
   "piexif==1.1.3", # this dep is a nightmare in terms of release management, better pinned just like in optimize-images anyway
-  "idna>=2.5,<4.0"
+  "idna>=2.5,<4.0",
+  "xxhash>=2.0,<4.0",
+  "types-xxhash>=2.0,<4.0",
 ]
 dynamic = ["authors", "classifiers", "keywords", "license", "version", "urls"]
 
diff --git a/src/zimscraperlib/zim/dedup.py b/src/zimscraperlib/zim/dedup.py
new file mode 100644
index 0000000..7f9f2cf
--- /dev/null
+++ b/src/zimscraperlib/zim/dedup.py
@@ -0,0 +1,80 @@
+import pathlib
+import re
+from typing import Any
+
+import xxhash
+from libzim.writer import Hint  # pyright: ignore[reportMissingModuleSource]
+
+from zimscraperlib.zim.creator import Creator
+
+CONTENT_BUFFER_READ_SIZE = 1048576  # 1M
+
+
+class Deduplicator:
+    """Automatically deduplicate potential ZIM items before adding them to the ZIM
+
+    This class automatically computes the digest of every item added to the ZIM, and
+    either add the entry (if item is not yet inside the ZIM) or an alias (if item with
+    same digest has already been added inside the ZIM).
+
+    This class must be configured with filters to specifiy which items paths to
+    consider. It is of course possible to consider all paths (i.e. all items) with a
+    wide regex or to operate on a subset (e.g. all images) with more precise filters.
+    Item is considered for deduplication if any filter matches. It is recommended to
+    properly configure these filters to save time / memory by automatically ignoring
+    items which are known to always be different and / or be too numerous.
+
+    Only the digest and path of items matching the filters are computed and stored.
+
+    The xxh32 algorithm (https://github.com/Cyan4973/xxHash) which is known to be good
+    at avoiding collision with minimal memory and CPU footprint is used, so the sheer
+    memory consumption will come from the paths we have to keep. This hashing algorithm
+    is not meant for security purpose since one might infer original content from
+    hashes, but this is not our use case.
+    """
+
+    def __init__(self, creator: Creator):
+        self.creator = creator
+        self.filters: list[re.Pattern[str]] = []
+        self.added_items: dict[bytes, str] = {}
+
+    def add_item_for(
+        self,
+        path: str,
+        title: str | None = None,
+        *,
+        fpath: pathlib.Path | None = None,
+        content: bytes | str | None = None,
+        **kwargs: Any,
+    ):
+        """Add an item at given path or an alias"""
+        existing_item = None
+        if any(_filter.match(path) is not None for _filter in self.filters):
+            if content:
+                digest = xxhash.xxh32(
+                    content.encode() if isinstance(content, str) else content
+                ).digest()
+            else:
+                if not fpath:
+                    raise Exception("Either content or fpath are mandatory")
+                xxh32 = xxhash.xxh32()
+                with open(fpath, "rb") as f:
+                    while True:
+                        data = f.read(CONTENT_BUFFER_READ_SIZE)  # read content in chunk
+                        if not data:
+                            break
+                        xxh32.update(data)
+                digest = xxh32.digest()
+
+            if existing_item := self.added_items.get(digest):
+                self.creator.add_alias(
+                    path,
+                    targetPath=existing_item,
+                    title=title or path,
+                    hints={Hint.FRONT_ARTICLE: True} if kwargs.get("is_front") else {},
+                )
+                return
+            else:
+                self.added_items[digest] = path
+
+        self.creator.add_item_for(path, title, fpath=fpath, content=content, **kwargs)
diff --git a/tests/zim/test_dedup.py b/tests/zim/test_dedup.py
new file mode 100644
index 0000000..1f36554
--- /dev/null
+++ b/tests/zim/test_dedup.py
@@ -0,0 +1,107 @@
+import pathlib
+import re
+from typing import Any
+
+import pytest
+
+from zimscraperlib.zim import Archive, Creator
+from zimscraperlib.zim.dedup import Deduplicator
+
+
+def test_deduplicator(
+    tmp_path: pathlib.Path,
+    png_image: pathlib.Path,
+    html_file: pathlib.Path,
+    html_str: str,
+    html_str_cn: str,
+):
+    main_path = "welcome"
+
+    png_data = png_image.read_bytes()
+
+    def add_items(creator_or_deduplicator: Any):
+        creator_or_deduplicator.add_item_for(
+            "welcome1", "wel1", content=html_str, is_front=True
+        )
+        creator_or_deduplicator.add_item_for(
+            "welcome2", "wel2", content=html_str, is_front=True
+        )
+        creator_or_deduplicator.add_item_for(
+            "dedup/welcome3", "wel3", content=html_str, is_front=True
+        )
+        creator_or_deduplicator.add_item_for(
+            "dedup/welcome4", "wel4", content=html_str, is_front=True
+        )
+        creator_or_deduplicator.add_item_for(
+            "prefix/dedup/welcome5", "wel5", content=html_str, is_front=True
+        )
+        creator_or_deduplicator.add_item_for("image1", None, fpath=png_image)
+        creator_or_deduplicator.add_item_for("image2", None, content=png_data)
+        creator_or_deduplicator.add_item_for("dedup/image3", None, fpath=png_image)
+        creator_or_deduplicator.add_item_for("dedup/image4", None, content=png_data)
+        creator_or_deduplicator.add_item_for("dedup/html", None, fpath=html_file)
+        creator_or_deduplicator.add_item_for("dedup/html_cn", None, content=html_str_cn)
+        creator_or_deduplicator.add_item_for(
+            "prefix/dedup/image5", None, content=png_data
+        )
+
+    fpath_without_dedup = tmp_path / "zim_without_dedup.zim"
+    with Creator(fpath_without_dedup, main_path).config_dev_metadata() as creator:
+        add_items(creator)
+
+    assert fpath_without_dedup.exists()
+
+    fpath_with_dedup = tmp_path / "zim_with_dedup.zim"
+    with Creator(fpath_with_dedup, main_path).config_dev_metadata() as creator:
+        deduplicator = Deduplicator(creator)
+        deduplicator.filters.append(re.compile("^foo/.*$"))
+        deduplicator.filters.append(re.compile("^dedup/.*$"))
+        deduplicator.filters.append(re.compile("^bar/.*$"))
+        add_items(deduplicator)
+
+        # added_items contains only original items, not the duplicates
+        assert set(deduplicator.added_items.values()) == {
+            "dedup/welcome3",
+            "dedup/image3",
+            "dedup/html_cn",
+        }
+
+    assert fpath_with_dedup.exists()
+
+    # check that deduplication has a consequence on ZIM size
+    assert (
+        fpath_without_dedup.lstat().st_size - fpath_with_dedup.lstat().st_size
+    ) > 3000  # 3291 as of libzim 9.3
+
+    for zim_path in [fpath_with_dedup, fpath_without_dedup]:
+        reader = Archive(zim_path)
+
+        assert reader.all_entry_count == 24
+
+        for html_path in [
+            "welcome1",
+            "welcome2",
+            "dedup/welcome3",
+            "dedup/welcome4",
+            "prefix/dedup/welcome5",
+            "dedup/html",
+        ]:
+            assert bytes(reader.get_item(html_path).content).decode() == html_str
+        assert bytes(reader.get_item("dedup/html_cn").content).decode() == html_str_cn
+
+        for img_path in [
+            "image1",
+            "image2",
+            "dedup/image3",
+            "dedup/image4",
+            "prefix/dedup/image5",
+        ]:
+            assert bytes(reader.get_item(img_path).content) == png_data
+
+
+def test_missing_content(tmp_path: pathlib.Path):
+    with Creator(tmp_path / "test.zin", "foo").config_dev_metadata() as creator:
+        deduplicator = Deduplicator(creator)
+        deduplicator.filters.append(re.compile(".*"))
+        with pytest.raises(Exception, match="Either content or fpath are mandatory"):
+            deduplicator.add_item_for("welcome", None)


Reviews:
user: login: rgaudin
body: LGTM ; thank you.

I appreciate that it's completely optional, explicit, independent and discrete.
We need to use it in the wild to see how it performs now.
state: APPROVED
author_association: MEMBER
submitted_at: 2025-05-08T10:47:46Z
user: login: benoit74
state: COMMENTED
author_association: COLLABORATOR
submitted_at: 2025-05-09T09:21:15Z
user: login: benoit74
state: COMMENTED
author_association: COLLABORATOR
submitted_at: 2025-05-09T09:26:58Z
user: login: benoit74
state: COMMENTED
author_association: COLLABORATOR
submitted_at: 2025-05-09T09:28:33Z

Review comments:
diff_hunk: @@ -0,0 +1,73 @@
+import pathlib
+import re
+from typing import Any
+
+import xxhash
+
+from zimscraperlib.zim.creator import Creator
+
+
+class Deduplicator:
+    """Automatically deduplicate potential ZIM items before adding them to the ZIM
+
+    This class automatically computes the digest of every item added to the ZIM, and
+    either add the entry (if item is not yet inside the ZIM) or a redirect (if item with
+    same digest has already been added inside the ZIM).
+
+    This class must be configured with filters to specifiy which items paths to
+    consider. It is of course possible to consider all paths (i.e. all items) with a
+    wide regex or to operate on a subset (e.g. all images) with more precise filters.
+
+    Only the digest/path of items matching the filters are computed and stored.
+
+    The xxh32 algorithm which is known to be good at avoiding collision with minimal
+    memory and CPU footprint is used, so the sheer memory consumption will come from the
+    paths we have to keep. This hashing algorithm is not meant for security purpose
+    since one might infer original content from hashes, but this is not our use case.
+    """
path: src/zimscraperlib/zim/dedup.py
user: login: rgaudin
body: Add how the filters are combined (here: any matching pattern triggers inclusion). That will save a look at the source for users
created_at: 2025-05-08T10:37:57Z
author_association: MEMBER
diff_hunk: @@ -0,0 +1,73 @@
+import pathlib
+import re
+from typing import Any
+
+import xxhash
+
+from zimscraperlib.zim.creator import Creator
+
+
+class Deduplicator:
+    """Automatically deduplicate potential ZIM items before adding them to the ZIM
+
+    This class automatically computes the digest of every item added to the ZIM, and
+    either add the entry (if item is not yet inside the ZIM) or a redirect (if item with
+    same digest has already been added inside the ZIM).
+
+    This class must be configured with filters to specifiy which items paths to
+    consider. It is of course possible to consider all paths (i.e. all items) with a
+    wide regex or to operate on a subset (e.g. all images) with more precise filters.
+
+    Only the digest/path of items matching the filters are computed and stored.
+
+    The xxh32 algorithm which is known to be good at avoiding collision with minimal
+    memory and CPU footprint is used, so the sheer memory consumption will come from the
+    paths we have to keep. This hashing algorithm is not meant for security purpose
+    since one might infer original content from hashes, but this is not our use case.
+    """
+
+    def __init__(self):
+        self.filters: list[re.Pattern[str]] = []
+        self.added_items: dict[bytes, str] = {}
+
+    def add_item_or_redirect(
+        self,
+        creator: Creator,
+        path: str,
+        *,
+        fpath: pathlib.Path | None = None,
+        content: bytes | str | None = None,
+        **kwargs: Any,
+    ):
+        """Add an item at given path or a redirect"""
+        existing_item = None
+        if any(_filter.match(path) is not None for _filter in self.filters):
+            if content:
+                digest = xxhash.xxh32(
+                    content.encode() if isinstance(content, str) else content
+                ).digest()
+            else:
+                if not fpath:
+                    raise Exception("Either content or fpath are mandatory")
+                xxh32 = xxhash.xxh32()
+                with open(fpath, "rb") as f:
+                    while True:
+                        data = f.read(65536)  # lets read stuff in 64kb chunks
path: src/zimscraperlib/zim/dedup.py
user: login: rgaudin
body: Given this cannot be parallelized and given the quantity of data a scraper handles, this seems **very** conservative (and thus very slow). I'd suggest we bump it ~10 times.
created_at: 2025-05-08T10:40:21Z
author_association: MEMBER
diff_hunk: @@ -0,0 +1,73 @@
+import pathlib
+import re
+from typing import Any
+
+import xxhash
+
+from zimscraperlib.zim.creator import Creator
+
+
+class Deduplicator:
+    """Automatically deduplicate potential ZIM items before adding them to the ZIM
+
+    This class automatically computes the digest of every item added to the ZIM, and
+    either add the entry (if item is not yet inside the ZIM) or a redirect (if item with
+    same digest has already been added inside the ZIM).
+
+    This class must be configured with filters to specifiy which items paths to
+    consider. It is of course possible to consider all paths (i.e. all items) with a
+    wide regex or to operate on a subset (e.g. all images) with more precise filters.
+
+    Only the digest/path of items matching the filters are computed and stored.
+
+    The xxh32 algorithm which is known to be good at avoiding collision with minimal
+    memory and CPU footprint is used, so the sheer memory consumption will come from the
+    paths we have to keep. This hashing algorithm is not meant for security purpose
+    since one might infer original content from hashes, but this is not our use case.
+    """
+
+    def __init__(self):
+        self.filters: list[re.Pattern[str]] = []
+        self.added_items: dict[bytes, str] = {}
+
+    def add_item_or_redirect(
+        self,
+        creator: Creator,
+        path: str,
+        *,
+        fpath: pathlib.Path | None = None,
+        content: bytes | str | None = None,
+        **kwargs: Any,
+    ):
+        """Add an item at given path or a redirect"""
+        existing_item = None
+        if any(_filter.match(path) is not None for _filter in self.filters):
+            if content:
+                digest = xxhash.xxh32(
+                    content.encode() if isinstance(content, str) else content
+                ).digest()
+            else:
+                if not fpath:
+                    raise Exception("Either content or fpath are mandatory")
+                xxh32 = xxhash.xxh32()
+                with open(fpath, "rb") as f:
+                    while True:
+                        data = f.read(65536)  # lets read stuff in 64kb chunks
+                        if not data:
+                            break
+                        xxh32.update(data)
+                digest = xxh32.digest()
+
+            if existing_item := self.added_items.get(digest):
+                creator.add_redirect(
path: src/zimscraperlib/zim/dedup.py
user: login: rgaudin
body: We should use aliases. We have no control over how those items are meant to be used and redirects handling is not transparent. Alias on the other end is.
created_at: 2025-05-08T10:42:10Z
author_association: MEMBER
diff_hunk: @@ -0,0 +1,73 @@
+import pathlib
+import re
+from typing import Any
+
+import xxhash
+
+from zimscraperlib.zim.creator import Creator
+
+
+class Deduplicator:
+    """Automatically deduplicate potential ZIM items before adding them to the ZIM
+
+    This class automatically computes the digest of every item added to the ZIM, and
+    either add the entry (if item is not yet inside the ZIM) or a redirect (if item with
+    same digest has already been added inside the ZIM).
+
+    This class must be configured with filters to specifiy which items paths to
+    consider. It is of course possible to consider all paths (i.e. all items) with a
+    wide regex or to operate on a subset (e.g. all images) with more precise filters.
+
+    Only the digest/path of items matching the filters are computed and stored.
+
+    The xxh32 algorithm which is known to be good at avoiding collision with minimal
+    memory and CPU footprint is used, so the sheer memory consumption will come from the
+    paths we have to keep. This hashing algorithm is not meant for security purpose
+    since one might infer original content from hashes, but this is not our use case.
+    """
+
+    def __init__(self):
+        self.filters: list[re.Pattern[str]] = []
+        self.added_items: dict[bytes, str] = {}
+
+    def add_item_or_redirect(
+        self,
+        creator: Creator,
path: src/zimscraperlib/zim/dedup.py
user: login: rgaudin
body: Why isn't this set on the Duplicator? Managing duplicates across multiple creators makes no sense and we have no multi-creator use case in sight anyway.
created_at: 2025-05-08T10:44:43Z
author_association: MEMBER
diff_hunk: @@ -0,0 +1,73 @@
+import pathlib
+import re
+from typing import Any
+
+import xxhash
+
+from zimscraperlib.zim.creator import Creator
+
+
+class Deduplicator:
+    """Automatically deduplicate potential ZIM items before adding them to the ZIM
+
+    This class automatically computes the digest of every item added to the ZIM, and
+    either add the entry (if item is not yet inside the ZIM) or a redirect (if item with
+    same digest has already been added inside the ZIM).
+
+    This class must be configured with filters to specifiy which items paths to
+    consider. It is of course possible to consider all paths (i.e. all items) with a
+    wide regex or to operate on a subset (e.g. all images) with more precise filters.
+
+    Only the digest/path of items matching the filters are computed and stored.
+
+    The xxh32 algorithm which is known to be good at avoiding collision with minimal
path: src/zimscraperlib/zim/dedup.py
user: login: rgaudin
body: Please include algo link as well: https://github.com/Cyan4973/xxHash
created_at: 2025-05-08T10:46:30Z
author_association: MEMBER
diff_hunk: @@ -0,0 +1,73 @@
+import pathlib
+import re
+from typing import Any
+
+import xxhash
+
+from zimscraperlib.zim.creator import Creator
+
+
+class Deduplicator:
+    """Automatically deduplicate potential ZIM items before adding them to the ZIM
+
+    This class automatically computes the digest of every item added to the ZIM, and
+    either add the entry (if item is not yet inside the ZIM) or a redirect (if item with
+    same digest has already been added inside the ZIM).
+
+    This class must be configured with filters to specifiy which items paths to
+    consider. It is of course possible to consider all paths (i.e. all items) with a
+    wide regex or to operate on a subset (e.g. all images) with more precise filters.
+
+    Only the digest/path of items matching the filters are computed and stored.
+
+    The xxh32 algorithm which is known to be good at avoiding collision with minimal
+    memory and CPU footprint is used, so the sheer memory consumption will come from the
+    paths we have to keep. This hashing algorithm is not meant for security purpose
+    since one might infer original content from hashes, but this is not our use case.
+    """
+
+    def __init__(self):
+        self.filters: list[re.Pattern[str]] = []
+        self.added_items: dict[bytes, str] = {}
+
+    def add_item_or_redirect(
+        self,
+        creator: Creator,
+        path: str,
+        *,
+        fpath: pathlib.Path | None = None,
+        content: bytes | str | None = None,
+        **kwargs: Any,
+    ):
+        """Add an item at given path or a redirect"""
+        existing_item = None
+        if any(_filter.match(path) is not None for _filter in self.filters):
+            if content:
+                digest = xxhash.xxh32(
+                    content.encode() if isinstance(content, str) else content
+                ).digest()
+            else:
+                if not fpath:
+                    raise Exception("Either content or fpath are mandatory")
+                xxh32 = xxhash.xxh32()
+                with open(fpath, "rb") as f:
+                    while True:
+                        data = f.read(65536)  # lets read stuff in 64kb chunks
path: src/zimscraperlib/zim/dedup.py
user: login: benoit74
body: Bumped to 1M for simplicity.
created_at: 2025-05-09T09:21:15Z
author_association: COLLABORATOR
diff_hunk: @@ -0,0 +1,73 @@
+import pathlib
+import re
+from typing import Any
+
+import xxhash
+
+from zimscraperlib.zim.creator import Creator
+
+
+class Deduplicator:
+    """Automatically deduplicate potential ZIM items before adding them to the ZIM
+
+    This class automatically computes the digest of every item added to the ZIM, and
+    either add the entry (if item is not yet inside the ZIM) or a redirect (if item with
+    same digest has already been added inside the ZIM).
+
+    This class must be configured with filters to specifiy which items paths to
+    consider. It is of course possible to consider all paths (i.e. all items) with a
+    wide regex or to operate on a subset (e.g. all images) with more precise filters.
+
+    Only the digest/path of items matching the filters are computed and stored.
+
+    The xxh32 algorithm which is known to be good at avoiding collision with minimal
+    memory and CPU footprint is used, so the sheer memory consumption will come from the
+    paths we have to keep. This hashing algorithm is not meant for security purpose
+    since one might infer original content from hashes, but this is not our use case.
+    """
+
+    def __init__(self):
+        self.filters: list[re.Pattern[str]] = []
+        self.added_items: dict[bytes, str] = {}
+
+    def add_item_or_redirect(
+        self,
+        creator: Creator,
+        path: str,
+        *,
+        fpath: pathlib.Path | None = None,
+        content: bytes | str | None = None,
+        **kwargs: Any,
+    ):
+        """Add an item at given path or a redirect"""
+        existing_item = None
+        if any(_filter.match(path) is not None for _filter in self.filters):
+            if content:
+                digest = xxhash.xxh32(
+                    content.encode() if isinstance(content, str) else content
+                ).digest()
+            else:
+                if not fpath:
+                    raise Exception("Either content or fpath are mandatory")
+                xxh32 = xxhash.xxh32()
+                with open(fpath, "rb") as f:
+                    while True:
+                        data = f.read(65536)  # lets read stuff in 64kb chunks
+                        if not data:
+                            break
+                        xxh32.update(data)
+                digest = xxh32.digest()
+
+            if existing_item := self.added_items.get(digest):
+                creator.add_redirect(
path: src/zimscraperlib/zim/dedup.py
user: login: benoit74
body: Fine, done
created_at: 2025-05-09T09:26:58Z
author_association: COLLABORATOR
diff_hunk: @@ -0,0 +1,73 @@
+import pathlib
+import re
+from typing import Any
+
+import xxhash
+
+from zimscraperlib.zim.creator import Creator
+
+
+class Deduplicator:
+    """Automatically deduplicate potential ZIM items before adding them to the ZIM
+
+    This class automatically computes the digest of every item added to the ZIM, and
+    either add the entry (if item is not yet inside the ZIM) or a redirect (if item with
+    same digest has already been added inside the ZIM).
+
+    This class must be configured with filters to specifiy which items paths to
+    consider. It is of course possible to consider all paths (i.e. all items) with a
+    wide regex or to operate on a subset (e.g. all images) with more precise filters.
+
+    Only the digest/path of items matching the filters are computed and stored.
+
+    The xxh32 algorithm which is known to be good at avoiding collision with minimal
+    memory and CPU footprint is used, so the sheer memory consumption will come from the
+    paths we have to keep. This hashing algorithm is not meant for security purpose
+    since one might infer original content from hashes, but this is not our use case.
+    """
+
+    def __init__(self):
+        self.filters: list[re.Pattern[str]] = []
+        self.added_items: dict[bytes, str] = {}
+
+    def add_item_or_redirect(
+        self,
+        creator: Creator,
path: src/zimscraperlib/zim/dedup.py
user: login: benoit74
body: Indeed, quite obvious, don't know how I missed this opportunity 🤣 
created_at: 2025-05-09T09:28:33Z
author_association: COLLABORATOR

PR comments:
user: login: benoit74
created_at: 2025-05-06T07:14:50Z
body: @rgaudin can you give me a first feedback on the proposed API. 

Code is still missing any test and versions of new libraries in pyproject.toml are wrong, but I prefer to first get feedback on the API since I'm really not convinced this is the optimal approach, but I fail to find something more convenient
author_association: COLLABORATOR
user: login: codecov[bot]
created_at: 2025-05-06T07:15:56Z
body: ## [Codecov](https://app.codecov.io/gh/openzim/python-scraperlib/pull/261?dropdown=coverage&src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=openzim) Report
All modified and coverable lines are covered by tests :white_check_mark:
> Project coverage is 100.00%. Comparing base [(`dc21be3`)](https://app.codecov.io/gh/openzim/python-scraperlib/commit/dc21be39cff3276871f594f8bbbbc85dde2320b2?dropdown=coverage&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=openzim) to head [(`f86b79c`)](https://app.codecov.io/gh/openzim/python-scraperlib/commit/f86b79ccb10e2c9d60ccccd06c08cef9ad1d04dc?dropdown=coverage&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=openzim).
> Report is 2 commits behind head on main.

<details><summary>Additional details and impacted files</summary>


```diff
@@            Coverage Diff            @@
##              main      #261   +/-   ##
=========================================
  Coverage   100.00%   100.00%           
=========================================
  Files           40        41    +1     
  Lines         2480      2512   +32     
  Branches       334       339    +5     
=========================================
+ Hits          2480      2512   +32     
```

</details>

[:umbrella: View full report in Codecov by Sentry](https://app.codecov.io/gh/openzim/python-scraperlib/pull/261?dropdown=coverage&src=pr&el=continue&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=openzim).   
:loudspeaker: Have feedback on the report? [Share it here](https://about.codecov.io/codecov-pr-comment-feedback/?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=openzim).

<details><summary> :rocket: New features to boost your workflow: </summary>

- :snowflake: [Test Analytics](https://docs.codecov.com/docs/test-analytics): Detect flaky tests, report on failures, and find test suite problems.
- :package: [JS Bundle Analysis](https://docs.codecov.com/docs/javascript-bundle-analysis): Save yourself from yourself by tracking and limiting bundle sizes in JS merges.
</details>
author_association: NONE
user: login: benoit74
created_at: 2025-05-06T07:21:34Z
body: And btw, should we prefer alias to redirect? It is still unclear to me when we should prefer one to the other
author_association: COLLABORATOR