Repository: internetarchive/openlibrary

Issue:
Add `messages.pot` updates to pre-commit hook (if possible)
<!-- IMPORTANT: Before posting, be sure to redact or remove sensitive data, such as passwords, secret keys, session cookies, etc. -->

<!-- A clear and concise description of what the problem is. Ex. I'm always frustrated when [...] -->


### Describe the problem that you'd like solved
<!-- A clear and concise description of what you want to happen. -->
It would be nice if the `messages.pot` file, which stores all the to-be-translated strings and their locations in the current version of the code, could be updated automatically with new PRs to keep everything more up to date for translators. After finishing up a bulk update to the `messages.pot` file for translations (#8817), I was talking with @cdrini about options for automating the process:

### Proposal & Constraints
<!-- What is the proposed solution / implementation? Is there a precedent of this approach succeeding elsewhere? -->
I'm not sure how the implementation for this would work, or whether it would be possible, but ideally one would bundle in the command that generates a new `messages.pot` file -- `docker compose run --rm -uroot home ./scripts/i18n-messages extract` -- into the already-existing `pre-commit` hook. More info re: updating in the [documentation](https://github.com/internetarchive/openlibrary/wiki/Internationalization-Contributor's-Guide-(i18n)#extracting-strings-from-htmlpython-files-generating-the-pot-file). 

This would be great, as it would mean that any time someone adds a new untranslated string, or modifies a file so the string line numbers are different, the update to the translation file would be included in the PR. Optionally/if possible, it would be great if the translation update could only run if the Python and/or HTML files had been modified, as those are the only changes that would affect it.

Alternatively, either if this isn't possible or just as a stop-gap measure, one could add a note to the Git Cheat Sheet that requests that people run the command themselves as part of the PR if their update has involved adding/changing any on-screen text. If this is the preferred solution or just something that would be good in the meantime, I'd be happy to do that update myself!

<!-- Which suggestions or requirements should be considered for how feature needs to appear or be implemented? -->

### Additional context
<!-- Add any other context or screenshots about the feature request here. -->

### Stakeholders
<!-- @ tag stakeholders of this bug -->
@cdrini 




Discussion:
I'm (provisionally) on it!!

Resolving patch:
diff --git a/openlibrary/i18n/__init__.py b/openlibrary/i18n/__init__.py
--- a/openlibrary/i18n/__init__.py
+++ b/openlibrary/i18n/__init__.py
@@ -1,8 +1,10 @@
 import os
 import shutil
 import sys
+import subprocess
 from collections.abc import Iterator
 from io import BytesIO
+from pathlib import Path
 
 import web
 
@@ -33,6 +35,23 @@ def warning_color_fn(text: str) -> str:
     return '\033[93m' + text + '\033[0m'
 
 
+def get_untracked_files(dirs: list[str], extensions: tuple[str, str] | str) -> set:
+    """Returns a set of all currently untracked files with specified extension(s)."""
+    untracked_files = {
+        Path(line)
+        for dir in dirs
+        for line in subprocess.run(
+            ['git', 'ls-files', '--others', '--exclude-standard', dir],
+            stdout=subprocess.PIPE,
+            text=True,
+            check=True,
+        ).stdout.split('\n')
+        if line.endswith(extensions)
+    }
+
+    return untracked_files
+
+
 def _compile_translation(po, mo):
     try:
         catalog = read_po(open(po, 'rb'))
@@ -139,11 +158,16 @@ def extract_templetor(fileobj, keywords, comment_tags, options):
     return extract_python(f, keywords, comment_tags, options)
 
 
-def extract_messages(dirs: list[str]):
+def extract_messages(dirs: list[str], verbose: bool, forced: bool):
     catalog = Catalog(project='Open Library', copyright_holder='Internet Archive')
     METHODS = [("**.py", "python"), ("**.html", "openlibrary.i18n:extract_templetor")]
     COMMENT_TAGS = ["NOTE:"]
 
+    untracked_files = get_untracked_files(dirs, ('.py', '.html'))
+    template = read_po((Path(root) / 'messages.pot').open('rb'))
+    msg_set = {msg.id for msg in template if msg.id != ''}
+    new_set = set()
+
     for d in dirs:
         extracted = extract_from_dir(
             d, METHODS, comment_tags=COMMENT_TAGS, strip_comment_tags=True
@@ -151,19 +175,30 @@ def extract_messages(dirs: list[str]):
 
         counts: dict[str, int] = {}
         for filename, lineno, message, comments, context in extracted:
+            file_path = Path(d) / filename
+            if file_path in untracked_files:
+                continue
+            new_set.add(message)
             counts[filename] = counts.get(filename, 0) + 1
             catalog.add(message, None, [(filename, lineno)], auto_comments=comments)
 
-        for filename, count in counts.items():
-            path = filename if d == filename else os.path.join(d, filename)
-            print(f"{count}\t{path}", file=sys.stderr)
+        if verbose:
+            for filename, count in counts.items():
+                path = filename if d == filename else os.path.join(d, filename)
+                print(f"{count}\t{path}", file=sys.stderr)
 
-    path = os.path.join(root, 'messages.pot')
-    f = open(path, 'wb')
-    write_po(f, catalog)
-    f.close()
+    has_changed = msg_set != new_set
+    if has_changed or forced:
+        path = os.path.join(root, 'messages.pot')
+        f = open(path, 'wb')
+        write_po(f, catalog, include_lineno=False)
+        f.close()
 
-    print('wrote template to', path)
+        print('Updated strings written to', path)
+    else:
+        print(
+            'No modified strings discovered. To force a template update, re-run with --force-write.'
+        )
 
 
 def compile_translations(locales: list[str]):


Test patch: