Repository: WeblateOrg/weblate

Issue:
test_ocr tests fails
### Describe the issue

When packaging weblate 4.14.2 for openSUSE test `test_ocr` fails almost always (more in the build server than on my workstation, but it is in both close to 100%):

```
[ 9019s] ======================================================================
[ 9019s] FAIL: test_ocr (weblate.screenshots.tests.ViewTest)
[ 9019s] ----------------------------------------------------------------------
[ 9019s] Traceback (most recent call last):
[ 9019s]   File "/home/abuild/rpmbuild/BUILD/Weblate-4.14.2/weblate/screenshots/tests.py", line 170, in test_ocr
[ 9019s]     self.assertIn('<a class="add-string', data["results"])
[ 9019s] AssertionError: '<a class="add-string' not found in '\n\n\n\n\n\n\n\n\n\n\n\n<table class="table table-condensed">\n  <thead>\n    <tr>\n      <th class="unit-state-cell"></th>\n      \n      \n      \n      \n      \n        \n          <th>\n        \n          English\n        \n        </th>\n      \n      \n        <th>Location</th>\n      \n      \n        <th>Assigned screenshots</th>\n      \n      \n        <th>Actions</th>\n      \n    </tr>\n  </thead>\n  \n    <tbody id="loading-screenshots" style="display:none;">\n      <tr><td colspan="5"><span class="icon-spin"  ><svg width="24" height="24" version="1.1" viewBox="0 0 24 24" xmlns="http://www.w3.org/2000/svg"><path d="M12,4V2A10,10 0 0,0 2,12H4A8,8 0 0,1 12,4Z"/></svg></span> Loading results…</td></tr>\n    </tbody>\n  \n  <tbody class="unit-listing-body">\n\n    \n      <tr class="warning">\n        <td colspan="5"><em>No matching strings found.</em></td>\n      </tr>\n    \n\n  </tbody>\n</table>\n\n'
[ 9019s] 
```

[Complete build log](https://github.com/WeblateOrg/weblate/files/10977793/_log.txt) with the list of all packages used and all steps taken to reproduce.


### I already tried

- [X] I've read and searched [the documentation](https://docs.weblate.org/).
- [X] I've searched for similar issues in this repository.

### Steps to reproduce the behavior

1. Run the test suite (see the attached log)

### Expected behavior

All test passing

### Screenshots

`test_ocr` fails almost always

### Exception traceback

```pytb
[ 9019s] ======================================================================
[ 9019s] FAIL: test_ocr (weblate.screenshots.tests.ViewTest)
[ 9019s] ----------------------------------------------------------------------
[ 9019s] Traceback (most recent call last):
[ 9019s]   File "/home/abuild/rpmbuild/BUILD/Weblate-4.14.2/weblate/screenshots/tests.py", line 170, in test_ocr
[ 9019s]     self.assertIn('<a class="add-string', data["results"])
[ 9019s] AssertionError: '<a class="add-string' not found in '\n\n\n\n\n\n\n\n\n\n\n\n<table class="table table-condensed">\n  <thead>\n    <tr>\n      <th class="unit-state-cell"></th>\n      \n      \n      \n      \n      \n        \n          <th>\n        \n          English\n        \n        </th>\n      \n      \n        <th>Location</th>\n      \n      \n        <th>Assigned screenshots</th>\n      \n      \n        <th>Actions</th>\n      \n    </tr>\n  </thead>\n  \n    <tbody id="loading-screenshots" style="display:none;">\n      <tr><td colspan="5"><span class="icon-spin"  ><svg width="24" height="24" version="1.1" viewBox="0 0 24 24" xmlns="http://www.w3.org/2000/svg"><path d="M12,4V2A10,10 0 0,0 2,12H4A8,8 0 0,1 12,4Z"/></svg></span> Loading results…</td></tr>\n    </tbody>\n  \n  <tbody class="unit-listing-body">\n\n    \n      <tr class="warning">\n        <td colspan="5"><em>No matching strings found.</em></td>\n      </tr>\n    \n\n  </tbody>\n</table>\n\n'
[ 9019s]
```


### How do you run Weblate?

Other

### Weblate versions

_No response_

### Weblate deploy checks

_No response_

### Additional context

_No response_


Discussion:
Do you have tesseract data installed on the test system? Maybe https://forums.opensuse.org/t/tesseract-ocr-wrong-data-directory/164659/9 is related?
This issue looks more like a support question than an issue. We strive to answer these reasonably fast, but [purchasing the support subscription](https://weblate.org/support/) is not only more responsible and faster for your business but also makes Weblate stronger.

In case your question is already answered, [making a donation](https://weblate.org/donate/) is the right way to say thank you!
> Do you have tesseract data installed on the test system? Maybe https://forums.opensuse.org/t/tesseract-ocr-wrong-data-directory/164659/9 is related?

Hmm, it is in `/usr/share/tessdata/eng.traineddata`, that should be correct, shouldn’t it?
This issue has been automatically marked as stale because there wasn’t any recent activity.

It will be closed soon if no further action occurs.

Thank you for your contributions!
This is not a support question! Does failure of your test suite not matter at all?
I think it's tesseract not working on your system, that's why I've tagged it this way.
What do you mean by “tesseract not working” for the insides of our build system?
It's not recognizing the text. It can be caused by wrongly installed data files, missing dependency on the tesseract data, or something else.

This test has been there for years (introduced in df4a52af1aa), it worked before on SUSE, it works anywhere else so far. All that makes me think the issue is in your environment, and not in Weblate.
Is there some command line command or something which could show me more what tesseract thinks is wrong with the setup?
`tesseract weblate/trans/tests/data/screenshot.png  -` in Weblate sources should produce something like:

```console
$ tesseract weblate/trans/tests/data/screenshot.png  -
Warning: Invalid resolution 0 dpi. Using 70 instead.
Estimating resolution as 133
Source string

Hello, world!+

One
Orangutan has %d banana.«!

Other
Orangutan has %d bananas.

Try Weblate at <http://demo.weblate.org/>!«!

Thank you for using Weblate.

Screenshot is shown to add visual context for all listed source strings.

 

```
This issue has been automatically marked as stale because there wasn’t any recent activity.

It will be closed soon if no further action occurs.

Thank you for your contributions!
That’s exactly (I believe) what I’ve got:
```
[   63s] + tesseract weblate/trans/tests/data/screenshot.png -
[   63s] Estimating resolution as 133
[   63s] Source string
[   63s] 
[   63s] Hello, world!+
[   63s] 
[   63s] One
[   63s] Orangutan has %d banana.<!
[   63s] 
[   63s] Other
[   63s] Orangutan has %d bananas.
[   63s] 
[   63s] Try Weblate at <http://demo.weblate.org/>!«!
[   63s] 
[   63s] Thank you for using Weblate.
[   63s] 
[   63s] Screenshot is shown to add visual context for all listed source strings.
[   63s] 
```

[Complete build log](https://github.com/WeblateOrg/weblate/files/11207526/_log.txt)

Actually, not exactly: `Orangutan has %d banana.«!` became `Orangutan has %d banana.<!` (a guillemet turning into less-than).
A few different characters should not matter. Anyway, this test is not really showing where the problem is, I will add a separate test for the OCR recognition itself so that the error is easier to diagnose.
Thank you for your report; the issue you have reported has just been fixed.

* In case you see a problem with the fix, please comment on this issue.
* In case you see a similar problem, please open a separate issue.
* If you are happy with the outcome, don’t hesitate to support Weblate by making a [donation](https://weblate.org/donate/).

Resolving patch:
diff --git a/weblate/screenshots/views.py b/weblate/screenshots/views.py
--- a/weblate/screenshots/views.py
+++ b/weblate/screenshots/views.py
@@ -2,8 +2,14 @@
 #
 # SPDX-License-Identifier: GPL-3.0-or-later
 
+from __future__ import annotations
+
 import difflib
+import os
+from contextlib import contextmanager
+from typing import TYPE_CHECKING
 
+import sentry_sdk
 from django.contrib.auth.decorators import login_required
 from django.core.exceptions import PermissionDenied
 from django.http import JsonResponse
@@ -14,21 +20,166 @@
 from django.views.generic import DetailView, ListView
 from PIL import Image
 
+from weblate.logger import LOGGER
 from weblate.screenshots.forms import ScreenshotEditForm, ScreenshotForm, SearchForm
 from weblate.screenshots.models import Screenshot
 from weblate.trans.models import Change, Unit
 from weblate.utils import messages
+from weblate.utils.data import data_dir
 from weblate.utils.locale import c_locale
+from weblate.utils.lock import WeblateLock
+from weblate.utils.requests import request
 from weblate.utils.search import parse_query
 from weblate.utils.views import ComponentViewMixin
 
-try:
-    with c_locale():
-        from tesserocr import RIL, PyTessBaseAPI
-    HAS_OCR = True
-except ImportError:
-    PyTessBaseAPI = None
-    HAS_OCR = False
+if TYPE_CHECKING:
+    from weblate.lang.models import Language
+
+with c_locale():
+    from tesserocr import OEM, PSM, RIL, PyTessBaseAPI, iterate_level
+
+TESSERACT_LANGUAGES = {
+    "af": "afr",  # Afrikaans
+    "am": "amh",  # Amharic
+    "ar": "ara",  # Arabic
+    "as": "asm",  # Assamese
+    "az": "aze",  # Azerbaijani
+    "az@Cyrl": "aze_cyrl",  # Azerbaijani - Cyrillic
+    "be": "bel",  # Belarusian
+    "bn": "ben",  # Bengali
+    "bo": "bod",  # Tibetan
+    "bs": "bos",  # Bosnian
+    "bg": "bul",  # Bulgarian
+    "ca": "cat",  # Catalan; Valencian
+    "ceb": "ceb",  # Cebuano
+    "cs": "ces",  # Czech
+    "zh_Hans": "chi_sim",  # Chinese - Simplified
+    "zh_Hant": "chi_tra",  # Chinese - Traditional
+    "chr": "chr",  # Cherokee
+    "cy": "cym",  # Welsh
+    "da": "dan",  # Danish
+    "de": "deu",  # German
+    "dz": "dzo",  # Dzongkha
+    "el": "ell",  # Greek, Modern (1453-)
+    "en": "eng",  # English
+    "enm": "enm",  # English, Middle (1100-1500)
+    "eo": "epo",  # Esperanto
+    "et": "est",  # Estonian
+    "eu": "eus",  # Basque
+    "fa": "fas",  # Persian
+    "fi": "fin",  # Finnish
+    "fr": "fra",  # French
+    "frk": "frk",  # German Fraktur
+    "frm": "frm",  # French, Middle (ca. 1400-1600)
+    "ga": "gle",  # Irish
+    "gl": "glg",  # Galician
+    "grc": "grc",  # Greek, Ancient (-1453)
+    "gu": "guj",  # Gujarati
+    "ht": "hat",  # Haitian; Haitian Creole
+    "he": "heb",  # Hebrew
+    "hi": "hin",  # Hindi
+    "hr": "hrv",  # Croatian
+    "hu": "hun",  # Hungarian
+    "iu": "iku",  # Inuktitut
+    "id": "ind",  # Indonesian
+    "is": "isl",  # Icelandic
+    "it": "ita",  # Italian
+    #    "": "ita_old",  # Italian - Old
+    "jv": "jav",  # Javanese
+    "ja": "jpn",  # Japanese
+    "kn": "kan",  # Kannada
+    "ka": "kat",  # Georgian
+    #    "": "kat_old",  # Georgian - Old
+    "kk": "kaz",  # Kazakh
+    "km": "khm",  # Central Khmer
+    "ky": "kir",  # Kirghiz; Kyrgyz
+    "ko": "kor",  # Korean
+    "ku": "kur",  # Kurdish
+    "lo": "lao",  # Lao
+    "la": "lat",  # Latin
+    "lv": "lav",  # Latvian
+    "lt": "lit",  # Lithuanian
+    "ml": "mal",  # Malayalam
+    "mr": "mar",  # Marathi
+    "mk": "mkd",  # Macedonian
+    "mt": "mlt",  # Maltese
+    "ms": "msa",  # Malay
+    "my": "mya",  # Burmese
+    "ne": "nep",  # Nepali
+    "nl": "nld",  # Dutch; Flemish
+    "nb_NO": "nor",  # Norwegian
+    #    "": "ori",  # Oriya
+    "pa": "pan",  # Panjabi; Punjabi
+    "pl": "pol",  # Polish
+    "pt": "por",  # Portuguese
+    "ps": "pus",  # Pushto; Pashto
+    "ro": "ron",  # Romanian; Moldavian; Moldovan
+    "ru": "rus",  # Russian
+    "sa": "san",  # Sanskrit
+    "si": "sin",  # Sinhala; Sinhalese
+    "sk": "slk",  # Slovak
+    "sl": "slv",  # Slovenian
+    "es": "spa",  # Spanish; Castilian
+    #    "": "spa_old",  # Spanish; Castilian - Old
+    "sq": "sqi",  # Albanian
+    "sr": "srp",  # Serbian
+    "sr_Latn": "srp_latn",  # Serbian - Latin
+    "sw": "swa",  # Swahili
+    "sv": "swe",  # Swedish
+    "syr": "syr",  # Syriac
+    "ta": "tam",  # Tamil
+    "te": "tel",  # Telugu
+    "tg": "tgk",  # Tajik
+    "tl": "tgl",  # Tagalog
+    "th": "tha",  # Thai
+    "ti": "tir",  # Tigrinya
+    "tr": "tur",  # Turkish
+    "ug": "uig",  # Uighur; Uyghur
+    "uk": "ukr",  # Ukrainian
+    "ur": "urd",  # Urdu
+    "uz_Latn": "uzb",  # Uzbek
+    "uz": "uzb_cyrl",  # Uzbek - Cyrillic
+    "vi": "vie",  # Vietnamese
+    "yi": "yid",  # Yiddish
+}
+
+TESSERACT_URL = "https://raw.githubusercontent.com/tesseract-ocr/tessdata_fast/main/{}"
+
+
+def ensure_tesseract_language(lang: str):
+    """
+    Ensure that tesseract trained data is present for a language.
+
+    It also always includes eng (English) and osd (Orientation and script detection).
+    """
+    tessdata = data_dir("cache", "tesseract")
+
+    # Operate with a lock held to avoid concurrent downloads
+    with WeblateLock(
+        data_dir("home"),
+        "screenshots:tesseract-download",
+        0,
+        "screenshots:tesseract-download",
+        timeout=600,
+    ), sentry_sdk.start_span(op="ocr.models"):
+        if not os.path.isdir(tessdata):
+            os.makedirs(tessdata)
+
+        for code in (lang, "eng", "osd"):
+            filename = f"{code}.traineddata"
+            full_name = os.path.join(tessdata, filename)
+            if os.path.exists(full_name):
+                continue
+
+            url = TESSERACT_URL.format(filename)
+
+            LOGGER.debug("downloading tesseract data %s", url)
+
+            with sentry_sdk.start_span(op="ocr.download", description=url):
+                response = request("GET", url, allow_redirects=True)
+
+            with open(full_name, "xb") as handle:
+                handle.write(response.content)
 
 
 def try_add_source(request, obj):
@@ -219,55 +370,70 @@ def search_source(request, pk):
     )
 
 
-def ocr_get_strings(api, image):
-    api.SetImage(image)
-    for item in api.GetComponentImages(RIL.TEXTLINE, True):
-        api.SetRectangle(item[1]["x"], item[1]["y"], item[1]["w"], item[1]["h"])
-        yield api.GetUTF8Text()
+def ocr_get_strings(api, image: str, resolution: int = 72):
+    api.SetImageFile(image)
+    api.SetSourceResolution(resolution)
+
+    with sentry_sdk.start_span(op="ocr.recognize", description=image):
+        api.Recognize()
+
+    with sentry_sdk.start_span(op="ocr.iterate", description=image):
+        iterator = api.GetIterator()
+        level = RIL.TEXTLINE
+        for r in iterate_level(iterator, level):
+            with sentry_sdk.start_span(op="ocr.text", description=image):
+                yield r.GetUTF8Text(level)
     api.Clear()
 
 
-def ocr_extract(api, image, strings):
+def ocr_extract(api, image: str, strings, resolution: int):
     """Extract closes matches from an image."""
-    for ocr_result in ocr_get_strings(api, image):
+    for ocr_result in ocr_get_strings(api, image, resolution):
         parts = [ocr_result, *ocr_result.split("|"), *ocr_result.split()]
         for part in parts:
             yield from difflib.get_close_matches(part, strings, cutoff=0.9)
 
 
+@contextmanager
+def get_tesseract(language: Language) -> PyTessBaseAPI:
+    # Get matching language
+    try:
+        tess_language = TESSERACT_LANGUAGES[language.code]
+    except KeyError:
+        try:
+            tess_language = TESSERACT_LANGUAGES[language.base_code]
+        except KeyError:
+            tess_language = "eng"
+
+    ensure_tesseract_language(tess_language)
+
+    with c_locale(), PyTessBaseAPI(
+        path=data_dir("cache", "tesseract") + "/",
+        psm=PSM.SPARSE_TEXT_OSD,
+        oem=OEM.LSTM_ONLY,
+        lang=tess_language,
+    ) as api:
+        yield api
+
+
 @login_required
 @require_POST
 def ocr_search(request, pk):
     obj = get_screenshot(request, pk)
-    if not HAS_OCR:
-        return search_results(request, 500, obj)
     translation = obj.translation
 
-    # Load image
-    original_image = Image.open(obj.image.path)
-    # Convert to greyscale
-    original_image = original_image.convert("L")
-    # Resize image (tesseract works best around 300dpi)
-    scaled_image = original_image.copy().resize(
-        [size * 4 for size in original_image.size], Image.BICUBIC
-    )
-
     # Find all our strings
     sources = dict(translation.unit_set.values_list("source", "pk"))
     strings = tuple(sources.keys())
 
     # Extract and match strings
-    with c_locale(), PyTessBaseAPI() as api:
+    with Image.open(obj.image.path), get_tesseract(translation.language) as api:
         results = {
             sources[match]
-            for image in (original_image, scaled_image)
-            for match in ocr_extract(api, image, strings)
+            for resolution in (72, 300)
+            for match in ocr_extract(api, obj.image.path, strings, resolution)
         }
 
-    # Close images
-    original_image.close()
-    scaled_image.close()
-
     return search_results(
         request, 200, obj, translation.unit_set.filter(pk__in=results)
     )
diff --git a/weblate/trans/context_processors.py b/weblate/trans/context_processors.py
--- a/weblate/trans/context_processors.py
+++ b/weblate/trans/context_processors.py
@@ -168,7 +168,6 @@ def weblate_context(request):
         "site_url": get_site_url(),
         "site_domain": get_site_domain(),
         "login_redirect_url": login_redirect_url,
-        "has_ocr": weblate.screenshots.views.HAS_OCR,
         "has_antispam": bool(settings.AKISMET_API_KEY),
         "has_sentry": bool(settings.SENTRY_DSN),
         "watched_projects": watched_projects,


Test patch:
diff --git a/weblate/screenshots/tests.py b/weblate/screenshots/tests.py
--- a/weblate/screenshots/tests.py
+++ b/weblate/screenshots/tests.py
@@ -7,22 +7,19 @@
 from difflib import get_close_matches
 from itertools import chain
 from shutil import copyfile
-from unittest import SkipTest
 
 from django.core.files import File
 from django.urls import reverse
 from django.utils import timezone
-from PIL import Image
 from rest_framework.test import APITestCase
 
-import weblate.screenshots.views
+from weblate.lang.models import Language
 from weblate.screenshots.models import Screenshot
-from weblate.screenshots.views import PyTessBaseAPI, ocr_get_strings
+from weblate.screenshots.views import get_tesseract, ocr_get_strings
 from weblate.trans.tests.test_models import RepoTestCase
 from weblate.trans.tests.test_views import FixtureTestCase
 from weblate.trans.tests.utils import create_test_user, get_test_file
 from weblate.utils.db import using_postgresql
-from weblate.utils.locale import c_locale
 
 TEST_SCREENSHOT = get_test_file("screenshot.png")
 
@@ -151,15 +148,9 @@ def test_source_manipulations(self):
         self.assertEqual(screenshot.units.count(), 0)
 
     def test_ocr_backend(self):
-        if not weblate.screenshots.views.HAS_OCR:
-            raise SkipTest("OCR not supported")
-
-        # Convert to greyscale
-        image = Image.open(TEST_SCREENSHOT).convert("L")
-
         # Extract strings
-        with c_locale(), PyTessBaseAPI() as api:
-            result = list(ocr_get_strings(api, image))
+        with get_tesseract(Language.objects.get(code="en")) as api:
+            result = list(ocr_get_strings(api, TEST_SCREENSHOT, 72))
 
         # Reverse logic would make sense here, but we want to use same order as in views.py
         matches = list(
@@ -174,8 +165,6 @@ def test_ocr_backend(self):
         )
 
     def test_ocr(self):
-        if not weblate.screenshots.views.HAS_OCR:
-            raise SkipTest("OCR not supported")
         self.make_manager()
         self.do_upload()
         screenshot = Screenshot.objects.all()[0]
@@ -194,24 +183,6 @@ def test_ocr(self):
             "OCR recognition not working, no recognized strings found",
         )
 
-    def test_ocr_disabled(self):
-        orig = weblate.screenshots.views.HAS_OCR
-        weblate.screenshots.views.HAS_OCR = False
-        try:
-            self.make_manager()
-            self.do_upload()
-            screenshot = Screenshot.objects.all()[0]
-
-            # Search for string
-            response = self.client.post(
-                reverse("screenshot-js-ocr", kwargs={"pk": screenshot.pk})
-            )
-            data = response.json()
-
-            self.assertEqual(data["responseCode"], 500)
-        finally:
-            weblate.screenshots.views.HAS_OCR = orig
-
     def test_translation_manipulations(self):
         self.make_manager()
         translation = self.component.translation_set.get(language_code="cs")
diff --git a/weblate/trans/tests/test_selenium.py b/weblate/trans/tests/test_selenium.py
--- a/weblate/trans/tests/test_selenium.py
+++ b/weblate/trans/tests/test_selenium.py
@@ -25,7 +25,6 @@
 )
 from selenium.webdriver.support.ui import Select, WebDriverWait
 
-import weblate.screenshots.views
 from weblate.fonts.tests.utils import FONT
 from weblate.lang.models import Language
 from weblate.trans.models import Change, Component, Project, Unit
@@ -524,11 +523,10 @@ def wait_search():
             element.submit()
 
         # Perform OCR
-        if weblate.screenshots.views.HAS_OCR:
-            self.click(htmlid="screenshots-auto")
-            wait_search()
+        self.click(htmlid="screenshots-auto")
+        wait_search()
 
-            self.screenshot("screenshot-ocr.png")
+        self.screenshot("screenshot-ocr.png")
 
         # Add string manually
         self.driver.find_element(By.ID, "search-input").send_keys(f"{text!r}")